How much discovery is enough?
More is better, but how much discovery is actually needed to get the job done?
One of the evergreen topics in product management remains product discovery. And probably for a good reason: Most product investments fail because they should not have been made in the first place. Either there never was a problem to begin with or the solution didn’t solve the problem.
But what makes product discovery so hard?
To begin with, recruiting customers for the research is not always easy. In one case, I only got sufficient customers to take time for in-depth customer interviews by giving out 75€ Amazon vouchers. It was always easier to talk to B2B customers as they tend to be more vested into the development of the products they use.
Nevertheless, a product manager’s schedule can be demanding, and carving out time for any proactive activities is sometimes a challenge. In these times and can simply feel overwhelming to do discovery right and half-assing might feel like a waste of time.
But any discovery is better than no discovery. And very often we over-estimate the amount of discovery necessary.
So let’s start with the basics.
The goal of discovery is to reduce uncertainty
When discussing discovery, we sometimes forget that the goal of discovery is to decrease uncertainty. There are several takeaways from that. First, it’s about reducing uncertainty. You will probably never eradicate uncertainty. Second, reducing uncertainty has value, while discovery activities create costs. These costs include the time invested in research activities and the costs of delay for not delivering a potential solution earlier.
This frame of mind helps you better prioritize what you want to discover. Product decisions are always like bets, meaning sometimes you lose. That’s part of the game. But some bets are so small that you should be sure not to over-invest to improve the odds.
If it is worth improving the odds, ask, “what part of an idea are you most uncertain about?” Investing time in these creates better returns than re-confirming things that can be considered best practices.
The challenge remains to correctly make the right assumptions about risks. You probably can’t continuously challenge the most underlying
Factors influencing the amount of discovery necessary
There are several other factors influencing the amount of discovery you actually need.
How certain are you about your ideal customer profile
Do you have a clear picture of your ideal customer? Maybe you even have quantitative evidence that a specific customer segment is especially valuable to your company e.g. as heavy users and loyal customers. Narrowing down the group of people whose problems and opinions really matter will increase the likelihood of representative results tremendously. If you want to create products for Botswanian fishermen, you should not ask American hairdressers for their opinion. If you target exclusively whales (e.g. fortune 500 companies) just gaining a single raving fan in that camp can be sufficient to continue. On the contrary, if your target audience is women between 20 - 50 years, you will probably need quite a while to get representative results.
How much variance is expected in an answer?
The easiest research questions are binary - Yes / No. E.g. simple a UX test where you simply test if a user understands how to use a function. But also when it comes to discovering if customers face specific problems, e.g. do you export this data to another software? This simple question might help you to identify interfaces to other software as a promising opportunity. Even with one sample, according to the "single sample majority interference, we can assume that in the binary case the likelihood that a single sample represent the majority is already at 75%!
If you are asking for assessments in an ordinal scale (e.g. NPS, CES) or even ratio scale (revenue, costs), theoretically, the answers can vary greatly. But the probability that the true median is within the range of responses of 5 samples is already at 93.75%. Let’s say you want to explore if you should automate the creation of delivery forms and you ask five customers how much time they spend daily creating these forms. The answers are 20, 25, 30, 32, and 40 minutes. Then we can be quite certain that the true median is somewhere between 20 and 40 minutes. This insight is only helpful if we observe a small spread. If the answer insight is “somewhere between 0 and 240 minutes” we might have to dig deeper. If the answers cluster quite closely within an attractive range, then you are good to go.How quickly do you start to see patterns?
Even in the case of very open questions (like my favorite UX research question “Tell me about what about your work annoys you most”) where you can possibly get an infinite amount of different answers, you might be surprised how quickly you hear the same answers over and over again. When I talked to fleet managers for FINN, most of them where complaining about being the “middle-men” between drivers and fleet provider. When I talked to people who were building houses for EuerZuhause, people had all kind of different challenges but the majority was desperately seeking for guidance. Sometimes it takes a few more interviews, but I feel like like it is usually not more than 10 until you start to see commonalities.
Quality and depth trumps volume
As with lot of trending topics most discovery activities I come across are shallow and of poor quality. When it comes to quality: Your research questions should not be suggestive, social desirable, platitudes, or ask for intentions.
Suggestive is anything that frames the customer towards an answer. Asking “Is X a good idea?” already frames X as good.
Social desirability is everything that is deemed to be morally good these days. “Is sustainability important for you” will more likely than not be answered with “yes” regardless if that person actually puts their money where their mouth is.
Platitudes are questions like “Are you interested in saving money?”, “Do you want to make more money?”, “Do you want to save time?”, “Is quality important for you”. Asked in these general terms you are likely to get zero insight from these question.
Lastly, asking for intentions is worthless. Never aks what people would do. If they would buy a product or would find certain features nice. These intentions are as reliable as new year resolutions. Ask for specific behaviors in the present or past instead.
When it comes to depth, sitting down with a single customer for an hour will give you a better idea of the motivations and reasons for their behavior. An email survey can provide lots of answers in a short time. But they lack depth. You rarely have the opportunity to really dig deep e.g. as to why a customer finds a specific feature useful or not.
Interview and automate
I strongly advise to use in depth customer interviews when creating a new product or if your are looking for product market fit. If your product is up and running and doing fine, you shouldn’t completely drop the interviews either. But the traction allows you to facilitate a more constant stream of feedback in the form of in product questionnaires, service tickets and analytics. All of these can provide evidence to confirm or disconfirm new ideas.
And best of all - you can automate them.
All things considered just a single good customer interview every week or two and some reliable sources for customer feedback can significantly decrease the uncertainty of your product investments.