This is the fourth article in a four-part series looking into how airlines manage the application of artificial intelligence in commercial functions.

Read part one: The Commercial Role in Artificial Intelligence – Lessons from Airlines

Read part two: Machine Learning & Artificial Intelligence in Airline Revenue Management: What’s the Goal?

Read part three: Managing Artificial Intelligence: Data Quality

Artificial intelligence has the potential to transform commercial airline operations: pricing, merchandising, scheduling, and more.

AI can tap into huge datasets to create reliable forecasts for customer behaviors and to develop algorithms for the optimal solution for maximizing revenue and customer satisfaction.

However, even with the possibilities for AI, there is no “perfect” algorithm. Given shortcomings in any such model, every AI/ML application requires active governance.

Airline revenue management, one of the first applications of Big Data and Machine Learning, has developed such a governance.

Perhaps RM experience offers both lessons – and warnings – for more recent applications of AI/ML – as AI sees more uses in airline commercial decisions and an even greater role in pricing.

“ModelOps” for AI – or the recommended framework for AI governance -- often begins with data quality, and then, only second, model explainability.

In airline RM explainability continues to be an issue even 50 years after the first RM models were deployed.  

Model Explainability

Model explainability in airline pricing systems exists conceptually – analysts generally understand how the model works.  However, the existing pricing models operate on a very micro basis – clusters of as few as five bookings on a specific flight on a specific date.

"Probably the most unintuitive factor for airline revenue management is treatment of demand uncertainty."

That can total potentially six million demand forecasts over the forecast period for an airline that operates 1,000 flights daily. In practice, it is impossible to dissect every output, every allocation of five or more seats on each flight (after all, that’s partly why we rely on the models).

Surely, most AI/ML models are similar – explainable conceptually but applied at such a micro level over so many transactions that it is difficult to explain the results in practice.

For airline revenue management, two common series of questions are:

  • Why does this high load factor flight not make money? Is the model holding enough seats for higher fare demand? Why can’t my high fare corporate customer get a seat?
  • Why does this flight consistently operate with empty seats? Is RM offering enough discount seats?

In these cases, at least one contributing factor may be demand uncertainty. In fact, probably the most unintuitive factor for airline revenue management is treatment of demand uncertainty: if the historic demand for a certain fare on a flight is highly volatile, the optimization algorithm allocates less inventory to it.

The “average” demand may be five passengers, but the allocation could be 1-2 or even no seats (often highly volatile demand is automatically collapsed into an adjacent demand grouping).  

The converse can happen too. Seats can be saved for very high fare demand – even if, on average, that demand is unlikely.

A $600 fare, for example, is worth 2-3 or more lower fare passengers. Insuring the flight can accommodate that very high fare demand – even if it’s a 50/50 probability -- is worth more than selling out all the seats at lower fares.

Distinguishing Model Error

Of course, another explanation for both empty seats and flights full of discount passengers could be….model error!  

Distinguishing model error from unintuitive results is often very difficult. Analysts may cite the general explanation to Sales or Finance in response to inquiries without fully acknowledging the possibility of model error.

"New AI/ML applications should include processes that can help users intervene when appropriate".

So, what can we learn from the RM experience with “model explainability?”  Basically, RM tells us that broad “model explainability” is only the first step in a three-step process. By itself, it is insufficient for optimal results. The three recommended steps are:

1. First, yes, gain that broad understanding. What inputs drive the results on both a macro and micro basis? What might drive unintuitive results? What is the basis for the AI/ML model’s customer segmentation scheme?

2. Second, introduce some sensible signals for model gaps. In RM, we track booking trends, knowing that the model is not likely to quickly adapt to a dramatic change and intervention may speed up the response.

We also may construct “red flags” for unusual behavior, like a low booked load factor three weeks in advance or a high booked load factor with low fare inventory still available.

One daily activity for analysts is to review the “red flags” highlighted the previous night. Finally, measuring and tracking forecast accuracy can signal model issues.

3. Third, listen to those closer to the actual markets who have real insight – even though the default should still be to rely on the model.

When Sales is persistent with complaints that seem to hold some logic, potentially test an alternative solution.  

For example, airline RM analysts today understand that the model won’t hold seats for highly uncertain demand but, in practice, when high fare demand is closed off in a particular market, analysts may choose to intervene anyway and insure there are seats for higher fare passengers should they materialize.

Even decades after their initial development, detailed model explainability in airline pricing can be a challenge. This will only get harder as models tap into more data inputs and more complex algorithms are developed.  

AI/ML model users should continue to insist on general “explainability” but they also need to be open-minded with respect to model gaps. New AI/ML applications should include processes that can help users intervene when appropriate.

Tom Bacon is an airline industry consultant based in Denver, Colorado. With 30 years experience in a variety of travel companies and business situations, he has a track record of dramatically improving profitability through innovative revenue strategies.