In a constantly changing industry, evolution of products and services is essential. Click. caught up with Lukas Vermeer, Director of Experimentation at Booking.com to discover how Booking.com uses controlled experimentation to conduct rapid, evidence-based product development.
Click.: Tell me about experimentation at Booking.com?
Vermeer: We run in excess of 1,000 concurrent experiments at any given moment across different products and target groups, allowing us to rapidly validate ideas and implementations. Each test is tailored towards a particular audience because we’re targeting specific solutions. This enables us to gather learnings on customer behaviours and discover what makes a meaningful difference for our users.
One way we do this is through A/B testing - something Booking.com is well known for - where we split the target audience into two groups. One half keeps the current version of a product, while the other half sees an updated version. We compare behaviours and if those using the new version behave as anticipated then we conclude the experiment worked as expected and put the change ‘live’ for everyone else. If it doesn’t work as assumed, then we’ve either misunderstood the problem or what the solution should be. At the very least, we’ve learned something and ensured we’re not making a change that doesn’t have the impact we want.
We don’t experiment because we like running experiments, but because experimentation is a great way to make sure that when we think we’re fixing something, we’re actually fixing it. Change is constant, we have to keep updating our products to make them better, but we also have to make sure those changes really work.
Click.: With so many experiments running at once, is there a risk of them overlapping?
Vermeer: Experiments can overlap, yes. In technical terms, you would say there’s a risk of interaction effects - two experiments having a different effect in isolation than when they are combined - but the ability to detect when this happens is actually an important benefit of experimentation. In product development it’s much more dangerous to isolate the experiments than to run them overlapping; you could test two changes that work independently, but if you isolate the tests you’ve no idea what’s going to happen when both are put live. We realise our overlapping approach has limits, but the intent is to build a better product, not perfect science.
Click.: What’s the importance of businesses challenging their assumptions?
Vermeer: If we make changes based purely on assumptions, we risk making unnecessary adjustments to our products. It’s crucial to check whether our changes actually have the desired effect on peoples’ behaviour. The worst thing we could do is constantly make all of these updates to our products, potentially confusing users if the changes don’t actually work how we intended.
One of the most important aspects of the entire scientific method and of experimentation is that you write assumptions down. There’s a practice we apply called pre-registration, where experiment owners specify from the start the assumptions they are making, how they are going to impact the customer behaviour and the metrics they will use to measure the experiment.
Click.: How can businesses empower employees to experiment?
Vermeer: This approach has always been ingrained in our company culture, with almost every change we make - no matter how big or small - wrapped in an experiment. I think two of the most important ingredients are transparency and accountability. We really encourage peer reviews - there needs to be some way for people to see each other's work and give their input. We’ve built this into our infrastructure by making descriptions of all iterations available, with the platform acting as a searchable repository of all previous experiments.
It’s also important for leaders to know that it’s okay to fail and be wrong - and communicate this to staff. When it comes to experimentation, people can sometimes be hesitant to adopt these methodologies. In my opinion, this is not a technical problem, but rather a mindset: you have to be willing to admit you don’t always know what’s best - and that’s okay. You need to be a little bit vulnerable and open to changing your mind. If instead you have a very set vision for what is right, and nothing could ever challenge the assumptions and opinions underpinning that vision, then effectively using experimentation is going to be problematic.
- Booking.com runs in excess of 1,000 concurrent experiments at any given moment across different products and target groups
- A/B testing involves a comparison of behaviours between two test groups, one that has been exposed to the new development, the other that remains with the original product
- Overlapping experiments can actually be beneficial as it allows for testing changes as they might interact in real life, rather than in isolation which does not reflect the 'live' situation
- It’s crucial to challenge assumptions and evaluate whether the changes actually have the desired effect on peoples’ behaviour
