Case Study: How stack ranking saved us from building the wrong product
80% of new features are rarely or never used. Back in May 2021, we almost fell victim to that statistic...
After a handful of identical requests from users in quick succession, we were on the verge of committing 3-months of development time to a huge new feature that we believed would fix a fatal flaw in our product funnel (spoiler, we were wrong).
This post explores the last minute experiment that saved us months of precious time and tens of thousands of Euro building the wrong thing.
An obvious solution to our funnel woes
In April 2021, after months of pivoting, we had finally landed on a proposition that was resonating with customers. Sign ups were surging and buzz was starting to build about our stack ranking tool that could help startups to validate their ideas and find product/market fit faster. But we had one major problem — people weren't sending their stack rank surveys out to participants.
Within just a few weeks, we started to hear the same feature request over and over again; our users wanted the ability to buy participants for their stack rank surveys.
These requests instantly felt reassuring to us, like 'Hey, this problem isn't to do with OUR product, it's just that these early-stage founders don't have users yet to participate in their stack rank surveys! 😅’
With this obvious solution handed right to us, we eagerly set off to get demos of the top tools for recruiting survey participants. Within a week, we had narrowed our list to the top 2 and were talking to their integration specialists about how we could build this feature into OpinionX.
Cue the cold feet...
The only thing that stopped us from moving ahead was the price tag — we needed to commit all our developer resources for at least 3-months to get this feature built. As an early stage startup, our vocabulary had always been in days or weeks. We had never committed to anything before that would require so much time.
We asked the integration specialists to give us a couple of days to confirm that we would definitely be continuing ahead with the project and we set about validating that this was the right project to work on. How would we validate that?
Well, we came to the conclusion that a feature demanding this much of our resources needed to solve the biggest problem our customers faced, otherwise we should be working on something else. It felt obvious to us that the right method to use was Customer Problem Stack Ranking — getting our users to vote on pairs of problem statements so that we could measure which ones were most important to them.
Starting our stack ranking experiment
Getting started was super easy; in under 30 minutes we wrote out the top 20 problems that users were experiencing — including a problem statement on the difficulty of finding participants — and added them to an OpinionX stack ranking survey. Then we grabbed the link and sent our stack ranking survey out to our most engaged users from the previous 6 weeks.
Over the following two days, users cast almost 800 votes and submitted 33 new problem statements to our stack ranking survey. It was immediately clear to us — not only was recruiting participants wasn't the most important problem, it wasn't even near the top of the stack rank.
Understanding the results
Digging into the data, we realized that we had blended together two completely different customer segments within our userbase: early-stage founders that were mostly pre-product and focused on idea validation, and product managers at scaling tech companies with existing userbases that were using OpinionX to prioritize their roadmaps and add quantitative data to their product discovery.
Once we split these two segments out, we realized that by not understanding the context of the feature request. Even if we did spend 3+ months building the panel recruitment integration, these early-stage founders didn't have the money to pay for participants and wouldn't end up using OpinionX often enough to become paying customers.
This stack ranking experiment didn't just help us to avoid working on the wrong feature. It also showed us the most important problems to be working on and gave us the first clue that we needed to improve our understanding of our ideal target customer.
What we learned:
1. Diagnose the problem that underpins each feature request.
People like to be nice. When they have a problem with your product, the easiest way to raise the issue with you is to dress that problem up as a potential solution. Feature requests are a route to understanding the barriers in your product, not a pipeline of ideas to fill your roadmap with.
2. Focus on the right customers.
When you've got a giant userbase, a rich product analytics dataset, and a skilled data analyst, behavioral segmentation is a breeze. That's not the case for most startups. We chased the wrong problem because we didn't realize who had that problem (and more specifically, who didn't).
Needs-based segmentation is vital — we've since added the ability to include multiple-choice questions in your stack rank survey so that you can segment your results to better focus on the customers that matter most.
3. Make decisions with data, not anecdotes.
Anecdotes and qualitative research are vital for understanding why people behave the way they do. But these methods also lead us astray so easily. Using a stack ranking tool like OpinionX gave us the best of both worlds — participants added new opinions to help us discover unknown problems and the stack ranking results gave us quantitative data for more informed decision making. Win win :)
Stack Ranking
Creating a similar stack ranking survey to the one we used in this project takes less than 5 minutes and is completely free. Before building that big new product or feature, check if it's really solving the most important problem that you should be focused on. Start building your first stack ranking survey for free here.