Common questions about Stack Ranking on OpinionX
“The five questions people ask before running their first stack ranking survey: how many participants they need, whether their opinions are written correctly, whether a long list makes the survey long, how to avoid biasing the results, and how participant-submitted opinions get into the ranking.”
Once you reach the end of this post, I guarantee you won't have a single worry left about stack ranking and how OpinionX works.
Here are the questions people ask most before running their first survey.
How many participants does a stack ranking survey need?
As a rule of thumb, take however many opinions you want to rank and multiply by four. Ten opinions means aiming for at least 40 participants.
That gets you a usable sample for a straightforward ranking survey. If your survey needs to hold up to scrutiny, or you're planning to segment the results, the proper calculation depends on your configuration instead of a flat multiplier. How to work out sample size for a ranking survey covers the formula.
Are my opinions written right?
Almost every time someone asks us this, their opinions are perfectly written.
When you're setting up the question, hit the suggestions button for best-practice examples. We'd also recommend a trial run with your team first, to get feedback on the sample opinions you wrote.
If you're staring at a blank list, problem brainstorming and writing problem statements both cover how to generate them.
If I add too many opinions, will the survey get much longer?
No.
OpinionX uses an algorithm to pick a sample of the opinions for each participant to vote on. That's a partial ranking, and it's why a survey with 100 opinions takes each participant about as long as one with 10.
That's the whole advantage of pairwise comparison over a drag-and-drop ranking question, where the mental work grows quadratically as the list gets longer.
How do I avoid biasing my survey?
As with any survey, make sure your question is neutrally written and doesn't carry an obvious point of view. Beyond that, two things help.
1. Include an open-response question. That lets participants submit new opinions to fill gaps you missed when setting up your list. You can add those to the voting list as you go, which closes your blind spots while the survey is still running. It's the clearest difference between this and a traditional survey.
2. Keep the tone and perspective consistent. Write all your opinions the same way. Make them all problem statements, instead of mixing problems together with ideas. A list that mixes the two is two questions pretending to be one. Assumption mapping is where that mixing usually creeps in.
Where do participants add new opinions, and how do I include them?
On your survey setup page, click the big plus button to add a new block, then use the dropdown to change it to an Open-Response question.
Once a participant submits an opinion at that step, you'll see it in your results and can use the buttons beside it to add it to the stack rank list right away, or edit it before including it.
This is worth doing on almost every survey. A large majority of stack ranks finish with at least one participant-submitted opinion in the top three, which means the most important problem is often one nobody on the team thought to include.
Didn't find your answer here?
Check the OpinionX Knowledge Base, where we've answered dozens of common questions about getting the most out of OpinionX. If you have a specific question, click the chat icon to message us, or grab a calendar slot to talk to someone from the team.
Otherwise, the best way to learn OpinionX is to jump in and try it. Create a stack ranking survey to test with your team. It only takes 2 minutes, and every question type and analysis feature is unlocked on the free tier, capped at 25 participants per survey.