Yource: Reducing 90% of claim handling time with human in the loop AI

Reducing 90% of claim handling time at tech scale-up Yource

Since 2011, Yource (also known as Vlucht-Vertraagd.nl) has been helping airline passengers successfully file claims to airlines for compensation for delayed, cancelled and overbooked flights. Every single day, up to 1,000 new claims are filed at their website. This ever growing pile of claims resulted in repetitive, time-consuming work for the claim handling experts of Yource. Together with the claim experts and the Yource IT team, Enjins developed the machine learning claim engine of Yource. Finally, leading to full automation of the claim validation task for nearly 90% of all claims.

Going from a manual process to 90% automation is a machine learning challenge, but even more an organizational challenge; a change management challenge. In this case study, we explain how we successfully managed to work towards this high level of automation and adoption of the new machine learning driven claim process by the employees of Yource.

Our approach consisted of 5 steps:

• Audit: Understanding the context of Yource and the claim process
• Model development: Gaining confidence in an offline setting
• Infrastructure setup: Productionizing machine learning models
• Silent pilot: Proving first results in a live setting
• Feedback loop: Human in the loop model deployment
• Automation: Full automation for claims with a prediction above the threshold

1. Audit: Understanding the context of Yource and the claim handling process

The business model of Yource is based on EU regulation that provides the right to customers to get compensated for delayed or cancelled flights

Regulation (EU) 261/2004: If your flight is delayed by three or more hours, cancelled or overbooked, and this is not the result of extraordinary circumstances, it entitles you to financial compensation.

Though this might sound like a binary rule, customers and airlines often interpret the ‘extraordinary circumstances’ differently. An airline will probably see heavy rain as an extraordinary circumstance, where a customer might have his doubts about this. Without going to court for every single case, Yource has to determine if the client rightfully submitted claims before they continue with their process of contacting the airline.

The Challenge in this process, as described by Mario Wester (CTO Of Yource):  

“In the past eight years we have expanded significantly, resulting in a rich and ever-growing dataset containing both claims and flight information. Our claim experts used this information in all steps of the process, but this was a time-consuming task. Our ambition was to speed up and partially automate the claim process. Therefore, we started cooperating with a Machine Learning company like Enjins.”

Mario continuous about the first step in the process, the claim validation:

“Our team at Yource comprises over 100 experts handling the claim process. The first step in this process is claim validation; the expert first checks if the customer’s claim is entitled for compensation. An expert needs 3-7 minutes for this process. Considering the growing amount of claims, experts spend most of their time on this repetitive and relatively easy step. Therefore, only limited time remains for the more complex tasks further in the claim process. Furthermore, the backlog grows, blocking our further growth. To realize further scalable growth of the company, we need to automate the validation step in the process.”

2. Model development: Gaining confidence in an offline setting

The main question in the modelling step was whether we could predict if a claim got positively or negatively validated. Using this prediction, we can decide to reject the customer or start the airline correspondence process. To develop the model and find the right data sources, talking to the claim experts was key. They did this process for multiple years, knowing exactly what variables they used (strike data, weather data, arrival times, etc) to make their decision. Since the company logged all decisions, a decent supervised machine learning trainset could be constructed. To assure expert involvement, the team used basic modelling techniques like decision trees. This allowed experts to provide feedback on the models already during the development stage.

3. Infrastructure setup: Productionizing machine learning models

Next to creating the ML models, Enjins created a data infrastructure to train, test and run models live in production. The team used Airflow for data orchestration. Within the AWS environment of Yource different services where setup (EC2, RDS and EKS). All model deployments and management were handled in Deeploy.

4. Silent pilot: Proving first results in a live setting

Though accuracy was solid (±85%) in an offline setting, we needed more proof before switching (even partly) from expert to model decision making. Therefore, we executed a silent pilot.  When a claim expert validated the claim, the claim was also sent to the machine learning service. In this way, we assured that the available data for the two ‘experts’ (the claim expert and the model) matched. At the end of every day, we compared the model and the expert. After a few iterations and improvements in mainly data availability of important features, accuracy also reached 80% in the live setting. This convinced the CEO to move towards decisions based on the ML model.

5. Feedback loop: Human in the loop model deployment

Now that trust is build, predictions were shown to the employees. Before going to full automation, employees got the opportunity to overwrite predictions of the model. For the cases (such as flights from a specific country) where no longer overrules were found full automation could be enabled. For the cases were the expert disagreed, valuable feedback was provided by the expert. Leading to for example adding new external data sources to feed relevant extra information to the model.

6. Automation: Full automation for claim handling with a prediction above the threshold

By using this feedback loop, constant improvement of the Machine Learning Engine was possible, leading to an increase in the percentage of fully automated cases.

Mario explains: “The claim validation that used to take an expert around 5 minutes is now handled in less than one second for nearly 90% of the claims. This saves our experts lots of time, making them available again for the more complex tasks.”

With a little help from our robot friends…

As mentioned in the beginning, finally achieving those great results was not only a technical challenge, but also a change management one. Ensuring that claim experts understood that they could now do more as a company, rather than being ‘scared’ of the AI development was key. The following two phrases of the CTO of Yource perfectly summarize this case study:

“We won’t need fewer people than we do now, we will just be able to process more claims with the same amount of people with a little help from our robot friends!”