Everyone hates bots, but the travel industry needs them

The airline industry is tightly braced for the ramp-up of air travel as the warmer months approach. And it needs every penny of revenue to compensate for colossal pandemic-inflicted losses.

In the effort to tackle that challenge, data-scraping bots that help operators streamline a massive real-time database of travel listings to optimize competitive pricing could be on the chopping block. They have taken a considerable toll on airlines, many of which have very humble profits on the actual flight tickets themselves. Most airlines’ revenue comes from in-flight upgrades and affiliate commissions on related bookings made through the airline website, such as car rentals or hotel accommodations.

The problem with eliminating bots, however, is they actually help travelers save by optimizing the booking process, making it easier and cheaper to find the deal most suitable for them.

Travel bots: Can’t live with them, can’t live without them

Despite the turbulence of the last few years, the travel industry remains a highly competitive business environment. And like many others, the industry has come to view data as a form of currency that holds the key to driving sales, reach and value.

It is perhaps to the detriment of the industry that online travel agencies such as Expedia, Kayak and Skyscanner realized this better and earlier than most, becoming the predominant force in today's travel space.

For most airlines, OTAs distribute and sell their flights, minimizing the losses on low occupancy. To do this, OTAs and aggregators scrape data about flight information from an airline under agreed-upon terms, in exchange for associated fees.

To scrape the data, OTAs use an automated script to access the airline or booking engine through an application programming interface (API), which is usually left open by the airline. Such scripts are otherwise known as bots and generally account for more than 90% of the traffic on an airline’s website.

Consequently, tickets aren’t booked directly with the airline, but with the OTA. The airline ends up sharing its negligent profit with the OTA, paying it an affiliation fee. Consider the absurdity for a moment: the airline pays the OTA for using the airline’s own data to make travelers book elsewhere. 

And since OTAs don't need to share data with airlines until check-in, they control much of the leverage if any changes occur throughout the customer process, while the airline is usually, and ironically, the one who ultimately shoulders the blame for customer travails.

This abundance of bot traffic creates another conundrum - it skews important business metrics designed for airlines to understand website and business performance, such as the look-to-book ratio, which is defined as the number of times a flight is requested per reservation made. But surely OTAs and airlines have combed out a better equilibrium by now, right?

Not exactly. As it has evolved, this open, scrape-friendly approach to driving business has exposed airlines to exploitation by both OTAs and malicious attackers alike. Collectively, from unauthorized scraping and seat spinning to loyalty programs’ misuse and the coordinated mass bookings of flight seats, bots create a significant headache for airlines. In 2017, only one industry – gambling - had a higher proportion of bad bots. And that’s before we even mention the struggles with fraud.

The question, then, is what to do about these bots.

Traditional bot mitigation does more harm than good 

Technology has gradually caught up to provide platforms with increasingly robust bot mitigation approaches, which wasn’t always the case.

Initially, security inspections like CAPTCHA were implemented with the aim of separating the human from the bot. Yet over time, these implementations have arguably done more harm than good, particularly with the user experience. Impatient customers began aborting purchases out of frustration, due to tedious security checks, while the users who make it through the CAPTCHA are often still, it turns out, bots. Ironically, they even started outperforming humans.

The need for a better equilibrium

Despite the minefield they create for security management, bots serve a crucial role in streamlining customer-friendly processes. As much as the industry reviles them, it relies on them in equal measure. The market needs a more harmonious equilibrium where humans get the most convenient and frictionless service and malicious bots are gradually weeded out - but not to the point of suffocating the OTA businesses. 

And the signs are encouraging. Today’s bot mitigation approaches and technologies are beginning to identify bad bots through improvements in detecting abnormal or suspicious behaviors. A promising strategy is to focus on the way the user is interacting with the web page and sending additional checkup challenges to users who display suspicious patterns. This is a good way to tell humans from bots, which is a prerequisite from telling good bots from bad bots.

Intensifying the focus on the user’s device can also prove valuable. Telemetry analysis can identify bot emulators designed to simulate real users more succinctly, while indicators such as whether a device has been “jailbroken” flag suspicious or illicit activity. 

This high-definition device fingerprint, instrumental in helping us understand certain characteristics of the user’s browser (header requests, browser extensions, use of graphic libraries) is becoming harder and harder for bot operators, or emulators, to spoof. 

Ironically, one of the most difficult things for these state-of-the-art bots to achieve is to emulate the randomness of real human users. Take cursor motion for example: A human pattern won’t ever have perfect straight lines, while bot patterns are all about those. Analyses of key presses and mouse movements are making real progress in this regard, but there’s still a long way to go.

Striking the right balance between safety and friction has traditionally been a game of give and take, but by using advances in technology and detection, platforms can start forging a path that takes the best of both worlds. 

This would be a most welcome move, since the airline industry has for too long pushed customers to the brink. As the relationship becomes healthier, such practices like airlines capitalizing on “calculated misery,” where they make their baseline products and services so low-quality and unpleasant that people will pay more to avoid them, can fortunately begin to dwindle.

For the airline industry, let's not forget the importance of safeguarding websites, mobile apps and APIs from malicious bots that love to disrupt its business. More importantly, let's remember that bots, whether bad or not, do not corrupt the ecosystem enough to the point where they damage both the airline and the customer. With the right innovation, travel and bots can peacefully coexist.