How Uber Eats experimentation culture led it to expand outside of food delivery

Daniele shares how Uber shifted from market research to directly testing assumptions and ideas in a real-world environment, and how it helped them to grow Uber Eats.

How Uber Eats experimentation culture led it to expand outside of food delivery

Uber Eats is an online food ordering and delivery platform launched by Uber in 2014.  It is operational in over 6,000 cities across 45 countries.

How does customer feedback influence your product decisions?

If I were to ask people five years ago, hey do you think Uber Eats should be delivering jewellery or clothing? Most people would have probably said no, no, that's not a product fit and here we are today delivering just that. Anything at the click of a button in the convenience of your own home or a location that you know where you need that specific thing.  

If we listened to our customers we would have just been a virtual food court. This is what Uber Eats was. It was a virtual food court, one place where you can get all your favourite brands, order your favourite food and get it delivered on time. But by not just listening to our customers, by running experiments, by testing new product features, by doing trials, we've actually become an entire virtual mall.

How do you distil insights from customers without asking them directly?

Be less dependent on market research and test your assumption and ideas in a real world environment. The first thing is get to know your users. What is the information that you have, the behaviours, the decisions, the choices that they make in their experience with your product? For us it could be when they order, what they order, what is their basket size, how frequently do they order, what are some of the things and the pain points and decision making points for your consumer in the journey.

The second thing is to protect the data. Keep personal information under lock. Make sure that is as safe as it can be, but give your team and the rest of the business access to information in line with legal parameters. The only way to truly create a data-driven culture is by giving them access to the information that they need in order to make better decisions that are data driven.

The number of problems and the number of unanswered questions that you have about your current users needs and your future users needs and behaviours just increases every single day.  

In order for you to truly understand them better, to really improve your product, your experiences, and that's just your business behaviour and performance overall, you really need to make experimentation a critical part of your strategy and execution. It shouldn't be something that's an afterthought or once a year or on big changes, but it should be entrenched in every single decision that you make.

Shifting towards more validation for ideas with experiments is great, but can be challenging to implement. How do you enable more frequent experimentation?  

In order to make experimentation entrenched in your business, you need the technology to support it. Running multiple experiments weekly at the same time is very time consuming. Focus on building technology platforms that allows teams to set up experiments very quickly. Make sure the technology not only allows them to set up the experiment, but also to run the results. If you have to ask your team to run these results manually every single time, it reduces their ability to run faster and create more experiments and ask more questions.

So build the tech that allows them to do that on every single product that you develop, on every single question you have, every engagement as quickly as possible, and you'll see how quickly you can learn.

The last fundamental is focus. When experimentation becomes entrenched in your company and when you have the technology that allows you to quickly execute and validate your assumptions, the world becomes your oyster. But as soon as the world becomes your oyster, the skill shifts to your ability to prioritise your assumptions. Make sure you are focused on only trying to test one thing at a time but run as many tests as possible. Focus on assumptions that could drive higher return on investments. This could be either from a financial perspective like conversion, or it could be from an experiential perspective. Make sure that there is one specific thing that you're focused on for that experiment and only try to validate that assumption.

Why does all of this matter?

In the past, acquisition was king. A company's ability to get a user on the platform was the most important thing, because the friction to move between products or companies was high. Most companies invest a lot of money in getting their customers onto the platform. So the most important thing was getting them onto the platform, getting them to use it, getting them to download that. And once they're there, they're sticky. But because in today's world and the technology environment that we work in, it's so easy for most consumers to jump between products, between companies, banks, things that were hard to move between a few years ago is a click of a button.

In today's life, retention is king. And the only way in which you can really drive and improve your retention is to drive loyalty, is to improve their experience, is to diversify your product, to create the light moments that they love, and also product updates. But you can only achieve this if you entrench and make experimentation part of your life in order to give your consumers updates and experiences and loyalty experiences that they didn't even know they want.

What advice do you have to help people shifting to more frequent experimentation?

The first guideline is that your objective of what you're trying to achieve and your hypothesis are two different things. Your objective needs to be linked to a business question that you're trying to answer or achieve or solve for. And your hypothesis should be linked to what you will be testing specifically. The other thing you want to do is spend time formulating your hypothesis. A lot of the time something that might seem very straightforward ends up being very complicated and something that might seem very complicated ends up being very straightforward. But if your hypothesis is clear and you know exactly what you are testing for, it is much easier to set up your entire experiment.

The second thing is, clearly define your primary and your secondary KPIs. A lot of the times in experiments, the results could show improvements in your primary KPIs. But where people get caught up is these unintended consequences. Ensure that you consider both what the good is that you were trying to achieve and what you might have achieved, as well as what might be some of the metrics that had a negative effect.

The other thing is to create a controlled environment as much as possible. This is a real world experiment. This is not a theoretic environment. So yes, exclude your treatment groups as much as possible from other elements and other engagements that could influence the results. But do not try to isolate them completely so that the results of your experiment are actually skewed and it's not really a true reflection of what the results would be if you had to make it live in a real world environment.

Also, think about the execution before you launch. The experimentations that you do need to be linked to something that you can change, implement and impact. If you test assumptions that provide incredible results, but you don't have the ability to deliver on them, you will actually be wasting your time. When it comes to the focus, make sure you're testing something that will drive business value as well as something that you can implement or that can impact your strategy in order to get the most from it.

Lastly, test one hypothesis at a time. You test one hypothesis using different treatments versus trying to test different hypotheses using one treatment. Really focus to ensure that you know what you do with the results, because this again will impact what this means for you going forward and the business results that you're trying to achieve.

What were the benefits of doing this for Uber Eats?

The conclusion of this experiment was that by giving people a choice between three options, instead of just giving them one, we got a higher uplift. The campaign drove a lot more efficient results. We created a gamified experience, which also increased our retention and engagement of the consumers that we targeted.  

This for me is another example of, and just a proof point of how important it is not just to make assumptions about what your product updates or initiatives will have on your trip averages or your clients' behaviour, but rather really embedding experimentation in your daily life in order to test your assumptions and really see what the impact of this would be on your business.