Pick The Right Ranking Method For Your Survey: 4 Simple Steps
You’re preparing to run a ranking survey but you’re worried that picking the wrong survey method could derail your entire project…
I’ve helped hundreds of researchers in your exact situation to design optimal ranking surveys that measure which ideas people like most, which features their users value most, or which problems their customers are most urgently trying to solve.
Here are the four simple questions I use to match the right ranking method to each research project…
Survey Formats For Ranking People’s Preferences
Before jumping to the four questions, it’s worth briefly clarifying -- what actually are ranking methods? There are five question types commonly used in online surveys for ranking people’s preferences:
Order Ranking
Points Allocation
Pairwise Comparison
MaxDiff Analysis
Conjoint Analysis
Each of these methods follow the same core research principle -- forced trade-offs. Instead of letting people rate each option individually out of 10 (ie. the flawed rating-scale question), comparative methods force people to choose between a set of options in a series of votes.
By adding everyone’s choices together, you get a relative importance score for each option, like this:
Pairwise comparison survey results on OpinionX ↗️
— — —
The Four Questions:
Q1. Single List or Categories?
Are you ranking a simple list of options or do your options belong to separate categories?
The ice cream flavors in List A can easily be compared and ranked against each other because they all belong to the same category. If your list of ranking options looks like this, jump ahead to Q2 →
The options in List B are choices you would consider when buying a new car, but the options in this list aren’t so easy to compare directly against each other. While we could vote on whether car color or car brand is more important to us, this list is asking us to compare stuff like Blue -v- Ford which is much harder because these options belong to separate categories. Even more importantly, every car must have a color and a brand, so comparing options across categories like this is counterproductive.
If your list of ranking options looks more like a list with multiple required categories like in List B above, then you should use Conjoint Analysis as your survey method…
Conjoint Analysis is a survey format designed for ranking options that belong to multiple categories. It shows sets of 2-6 profiles, where profiles all share the same categories but each profile shows different options within those categories (eg. Blue Gas-Powered Ford vs Black Electric Tesla). By randomizing the options shown on these profiles after each vote, we can see which categories are most important and which options are most preferred within each category.
Below you’ll see an example of a conjoint analysis survey, where a product team at a software company is measuring which features their customers perceive as being most valuable:
Conjoint analysis survey created for free on OpinionX ↗️
— — —
Q2. List Length?
How many options are you trying to compare and rank?
If you’re only ranking a short list of 3-10 options, like in List A, then continue on to Q3.
If you’re trying to rank a list that contains more than 10 options, like List B, then jump ahead to Q4 →
— — —
Q3. Equal or Skewed Input?
Should everyone have an equal say in picking one winner or should stronger opinions carry more weight?
Since your list only includes 3–10 items, it’s short enough for participants to evaluate the whole list without it being overwhelming for them. That leaves you with two voting methods → Order Ranking and Points Allocation.
Order Ranking asks respondents to rank the entire list from most to least preferred, producing a clean list of priorities without ties in each person’s ranking. It captures relative position of preference -- ie. what comes first, second, third, etc for each person. Since everyone’s rankings are treated equally, Order Ranking is ideal when your goal is to choose a single winner.
Order Ranking survey created for free on OpinionX ↗️
Points Allocation gives respondents a pool of points to distribute freely across your list of options. It captures the relative weight of preference -- ie. not just which options people prefer, but how much they care about each one. This approach lets participants emphasize what matters most and skip what doesn’t, making it better suited to situations where you’re picking multiple winners.
Points Allocation ranking survey created for free on OpinionX ↗️
Here’s an example to help demonstrate these differences…
List A is testing potential podcast names. I’m looking for the single best option and I want everyone’s input to be weighed equally. By forcing a consistent voting format for each respondent, Order Ranking will do a better job of surfacing the most broadly appealing name.
List B is evaluating customer pain points to help prioritize our product roadmap. Here, I want to identify spikey problems that elicit strong opinions from people, even if that comes from a small segment of customers shouting loudly about a subset of options. Points Allocation is more suitable because it lets respondents skip irrelevant issues and throw more weight behind what really matters to them. Since we’ll be choosing the 2-3 top-ranked problems to investigate further, I care less about giving everyone equal input and more about where opinions are loudest.
— — —
Q4. Complex Options?
How easy is it for participants to quickly understand each of your ranking options?
When your ranking list has more than 10 options, showing them all on screen at the same time can become an overwhelming task for people. Have you ever tried to rank a list of 20 options on a mobile phone? It’s not fun.
Almost 60% of surveys are completed on phone screens today. That’s a key reason why Matrix Grid questions have almost entirely disappeared from online surveys; down from almost 50% of all surveys in 2015 to just 19% of surveys in 2020. There are a lot of easier ways to collect ranked preferences for long lists via mobile phones. Let’s look at two.
Pairwise Comparison shows your list of options as a series of head-to-head votes. Options are given a preference score based on the percentage of pairs they’re picked as the winning choice.
Pairwise Comparison survey created for free on OpinionX ↗️
MaxDiff Analysis shows 3-6 options at a time and asks respondents to pick the ‘best’ and ‘worst’ options in each set. By subtracting the percentage of bests from worsts, you can rank the list of options by overall preference.
MaxDiff Analysis survey created for free on OpinionX ↗️
Both of these research methods are excellent for ranking long lists of options. They’re both flexible and can be tailored to various use cases by rewording the question or voting labels -- for example, change ‘best’ and ‘worst’ on MaxDiff Analysis to ‘Most Important’ and ‘Least Important’.
However, I don’t use Pairwise Comparison and MaxDiff Analysis completely interchangeably -- here are some ways determine which format is a better fit for each individual survey…
i. Complexity
Are you ranking short statements that can be understood at a glance? Showing 3-6 long complex statements on screen multiple times in a row can be overwhelming, which makes MaxDiff Analysis a poor fit. For longer complex statements, I generally prefer to keep things simple and tidy by choosing Pairwise Comparison instead.
ii. Population
How many people are available to complete your survey? If you’re participant constrained with a small group of voters, then MaxDiff Analysis is the better option -- it gathers preference data much faster than Pairwise Comparison by showing 3-6 options per set instead of just 2.
iii. Perspective
MaxDiff Analysis balances ‘best’ and ‘worst’ votes to calculate a preference score for each option. But sometimes, I’m not interested in ‘worst’ votes at all -- if I’m ranking problem statements, I’m only interested in knowing which problems people are most frustrated by and I don’t care which ones they consider low priority. MaxDiff Analysis balances people’s perspectives, while Pairwise Comparison focuses solely on your ranking criteria. Choose accordingly!