The Business – Since 2011, Yource (also known as Vlucht-Vertraagd.nl) has been helping airline passengers successfully file claims for compensation for delayed, cancelled and overbooked flights. Every single day, up to a 1,000 new claims are filed at their website.
The Challenge– In the past eight years Yource has expanded significantly, resulting in a rich and ever-growing dataset containing both claims and flight information. This information is used by experts in all steps of the process. Yource aims to speed up and partially automate the claim process using Machine Learning.
The Yource team consists of over 100 experts handling the claim process. The first step in this process is claim validation; the expert does a first check on whether the customer’s claim is entitled for compensation. An expert needs 3-7 minutes for this process. Considering the growing amount of claims, experts spend most of their time on this repetitive and relatively easy step in the process. Therefore, only limited time is left for the more complex tasks further in the claim process. Furthermore, the backlog grows, blocking further growth. To realize further scalable growth of the company, we need to automate the validation step in the process.
The main question in this step is whether we can predict if a certain claim is going to be successful or not. When experts make this decision manual, they use several internal and external information. To get all these data sources together and store them in one place, a scalable and future-proof data infrastructure is needed. The overall goal is to get Machine Learning in production within the existing technology stack at Yource.
The Solutions– Learning from the experts, we got to understand the whole process and existing technology at Yource. We worked together closely with the business and development teams to build production-ready ML engines to automate the claim processes. The claim validation that used to take an expert around 5 minutes is now handled in less than one second for the majority of the claims.
Next to creating the ML models, we set up the data infrastructure to train and run models live in production. The data infrastructure can be extended easily and new models can be trained, deployed and used in production. In this way, we automated multiple steps in the claim process.
Before machine learning takes over (most of the) manual work, it is crucial to run pilots and take feedback from experts into account. We took several steps to work towards production:
- Starting with a silent pilot, in which no employee knows that live predictions of their decisions are made in the background.
- We run pilots and create a constant feedback loop. Employees can overwrite the predictions of the model and this way give live feedback on the performance of the model. Leading to for example adding new external data sources to feed extra relevant information to the model. By using this feedback loop, constant improvement of the Machine Learning Engine is possible.
- Finally, after extensive testing, the ML models run in production and take over manual jobs.
To have quick insights in the performance of the model and flows in the automated process, we built the Enjins ML Monitor. This is a dashboard in which several graphs and important KPIs are shown and errors can be detected and handled quickly.