How to Innovate with Quick Experiments

I see a lot of building. Build, build, build. And in that process, people are really falling into what I call the product discovery valley of death. And this is where you have great ideas, but you don't test them.

How to Innovate with Quick Experiments

Product Discovery Group works with product teams as a mentor and Product Discovery Coach to build confidence in their ideas more quickly using less engineering effort. They engage clients and their customers in idea development, user testing, user research, and developing rapid prototypes.

What is the biggest problem that you are seeing with the organisations that you are coaching for?

I see a lot of building. Build, build, build. And in that process, people are really falling into what I call the product discovery valley of death. And this is where you have great ideas, but you don't test them. And when you don't test them, there's the risk of people not adopting them, not seeing value. You may have great ideas. You may have interviewed folks, but you haven't taken the solutions back to those folks to really see if they would use them. And today, today it's all about these quick experiments for innovation so that when you do this solution-testing step then you can avoid the valley of death.

That is a relatable problem. But how do companies avoid the build trap?

Companies need to innovate with quick experiments. The first step, the first word, is innovate. For me, this includes two key elements: getting the right team, and getting the best ideas from the team.  

When I say the right team I mean that you really need a cross-functional team. This is important. As you get more folks on the team earlier in the process and you solicit their ideas, you're going to get a much richer set of ideas to start from. That might be in the problem space. That might be in the solution space, but you need this cross-functional team. I have clients that have a data scientist on every cross-functional team. I have clients that have doctors on every cross-functional team. So sometimes you need subject matter experts, sometimes data experts, somebody from customer support maybe.  

To get the best ideas from the team I want people to draw it, not just a designer. All of the team should draw. These are things that drive innovation when you have people write down their ideas and then you take these ideas to your users in lightweight forms. The Sprint book is a great resource for this type of activity. So I recommend checking this out. You don't need to do these things in five days. You can just take these exercises and apply them every sprint, every discovery cycle.

What are the challenges that you see when teams are trying to innovate?

Most common problem I see is that folks create cross-functional teams, but don't drive engineer participation. And so what they're losing is what's just now possible. Really engineers are one of our greatest sources of innovation. They can hear the problem and they can apply perhaps a new solution they've been thinking about or reading about or one that might just come out of what their current workstream is. So the concept of just now being possible is really important for teams. You might be missing this if you don't have engineers.

A lot of the most commonly used companies today had either all or more than half engineer founders. It's really important to make sure that you don't lose touch with your engineers. Keep them involved. Amazon took some just now possible technology, secure websites, widespread internet access to launch in the 90s. My company Power Reviews, we were a product reviews widget inside of the site. We grew to 1,200 clients in 20 countries, all powered by iFrames and Ajax. Tesla really started with this concept of a long-lasting battery.  

Innovation is critical. But how do you move past the idea phase?

Let’s use an example of a backlog prioritisation experiment. I want you to pretend that you work for Zoom and that you have thought of a bunch of ideas for improving breakout rooms. I can tell you that the various frameworks for prioritisation aren't that useful because you're often making up numbers for what's the impact, what's the confidence, what's the effort. If you do some discovery about the ideas, do some user testing, you can learn about the priority with respect to your customer's value. Do they want this thing? That's really what we're interested in. So let's go talk to users!

One experiment is that we could start sending push messages to users to gauge their interest. I call these micro prototypes. An examples could be:

“Zoom now offers advanced features to manage the success and health of breakout rooms.”  

I can gauge whether somebody's interested in breakout room improvements right away. Would they swipe to learn? Would they dismiss? Have them talk about it in front of you in a moderated interview.


Alternatively we could show different messages to quickly validate other ideas.

“Zoom now offers the ability to listen into a breakout room without having to enter it.”

“A new feature, see who has video turned on or off in breakout rooms without having to enter them.”

“New feature, for breakout rooms Zoom shows you which rooms are the quietest and which rooms have the most people talking.”

Again, you may not be the product manager for Zoom breakout rooms or Zoom itself but what I'm encouraging you to do is to think of the ideas in your backlog and quick ways to explain them to users and to get their interest.  

The results of the quick experiments help you to decide what to work on. I did this with a client. They worked with educational technology. We went into a classroom and we showed a variety of ideas and the team was able to hear the excitement and understand the response as to which of these ideas was most valuable.  

What are the challenges that you see with quick experiments?

What do I see in the market? Lots of stuff. People testing too much, creating these end-to-end experiences. So what I want you to do, I want you to not test an entire experience. I want you to pick a piece of that experience to innovate. This is an e-commerce flow. How about just picking add to cart? And really testing something.

So again, we can be quicker if we can focus. You can certainly test each of these other aspects, but to be quick, you want to test them one at a time, not in these mega testing sessions. I also want you to start with a vertical slice. So what is really valuable about that add to cart area? Can you launch something? Can you from end to end deliver a piece of value quickly? So when I say quick, you may have a vision of that entire product that is reflected in that left pyramid that's full. What I really want you to do is iterate these vertical slices.


How do you design good experiments to test the ideas?

Innovation can often be a quantity game. When you offer no choices to users, effectively you're saying, hey, choose this thing. Experiments help you avoid confirmation bias. When you offer choices, you're being open-minded. And as you're open-minded, the idea from that data scientist might really come through as the winner or the doctor that you put on the team. Doesn't have to be the product manager. Doesn't have to be the designer. In fact, the engineer might come up with the idea that the user selects.

Another issue that you will face is “blah” feedback. “I like this”. “That's interesting”. This is all kind of meh feedback. This is not what you want. People are being nice to you. I want to hear, “ooh”. I want to hear, “oh, wow”. I want to hear, “is this available?” I want that emotional reaction.

What outcomes have you seen from putting this into practice?

I worked with an entrepreneur who spent $200,000 building a website that nobody went to. And he had found an outsourced company to build his dream idea. But the issue with the dream idea is that he wasn't able to market it and didn't quite understand what people would see value in it. It was valuable to him, but he hadn't done the research with others.  

And so when he came to me, we started from scratch. We built some prototypes to test what people would see on this website. And I'll tell you what this website does. Imagine that you have a home project and you need a plumber or an electrician. While you have to go find somebody, you might be doing it online. A lot of people ask their friends or their family. The premise of this site was that you would ask your neighbours. And so how can you facilitate the asking of neighbours? Will you have to find them? What neighbourhood are you in? And we tested the interest in this idea and we tested how people could find out about it. And what we learned led us to create a very bare-bones, no-code website.

We started with fake prototypes, quick experiments, moved to the next set of quick experiments, which was to use a Squarespace site with Google spreadsheets and to drive traffic to that site from Nextdoor and Facebook groups and you can imagine Squarespace plus Google Sheets is much cheaper than a $200,000 site custom. We'd learned about Nextdoor and Facebook as a large source of these plumber and electrician referrals from our customer interviews. When this site got too overloaded and we started to actually drive traffic we went on to a site using Webflow, another sort of more advanced no code tool.  

Then he also decided to pivot his idea into a home service subscription service. So for plumber and electrician, whatever you needed, what about a monthly fee to have that taken care of? And so through these quick experiments, this entrepreneur was able to pivot and then grow his understanding of his market. And this was in a fraction of the time it took to build this other site that no traffic was going to.  

So again, quick experiments can help you innovate to get to the right idea. And so this is really what we're looking for. Instead of building all the time, it's learning, building, and then measuring.