Pick The Right Ranking Method For Your Survey: 4 Simple Steps
There are five survey ranking methods for ordering people's preferences: order ranking, points allocation, pairwise comparison, MaxDiff analysis and conjoint analysis.
“Four questions decide which one you need. Are your options a single list or do they belong to separate categories (categories mean conjoint)? Is the list short (3 to 10) or long (more than 10)? Should everyone have equal input, meaning order ranking, or should stronger opinions carry more weight, meaning points allocation? And for long lists, are the options simple enough to compare at a glance (MaxDiff) or long and complex (pairwise comparison)?”
Every one of these methods forces a trade-off, which is what makes them more reliable than a rating scale where people score everything highly and tell you nothing.
What are survey ranking methods?
A survey ranking method is a question format that measures people's preferences by making them compare options against each other, rather than rating each option on its own. These methods are a form of ordinal ranking built for surveys.
There are five commonly used in online surveys:
Each follows the same core principle: forced trade-offs. Instead of letting people rate each option individually out of ten, which is the flawed rating-scale question built on a Likert scale, comparative methods make people choose between options across a series of votes. Add everyone's choices together and you get a relative importance score for each option.
Pairwise comparison survey results on OpinionX ↗️
The reason this matters is that a rating scale doesn't force anyone to decide. Ask people to score twenty options out of ten and most land between six and eight, which tells you nothing about what they'd actually prioritise. A ranking method makes them separate the options.
Which survey ranking method should you use?
Four questions get you to the right method. This table is the whole decision in one screen; the sections below walk through each question.
| If your situation is... | Use this method |
|---|---|
| Options belong to separate categories | Conjoint analysis |
| Short list (3–10), picking one winner, equal input | Order ranking |
| Short list (3–10), picking several, stronger opinions weigh more | Points allocation |
| Long list (10+), short simple options | MaxDiff analysis |
| Long list (10+), long complex options | Pairwise comparison |
Q1. Single list or categories?
Are you ranking a simple list of options, or do your options belong to separate categories?
The ice cream flavours in List A can be compared and ranked against each other because they all belong to the same category. If your list looks like that, jump ahead to Q2.
The options in List B are choices you'd consider when buying a new car, but they aren't easy to compare directly. We could vote on whether car colour or car brand matters more, but this list asks us to compare things like "Blue vs Ford", which is much harder because those options belong to separate categories. More to the point, every car must have both a colour and a brand, so comparing options across categories is counterproductive.
If your list looks like List B, with multiple required categories, use conjoint analysis.
Conjoint analysis is designed for ranking options that belong to multiple categories. It shows sets of two to six profiles, where every profile shares the same categories but shows different options within them (for example, Blue Gas-Powered Ford vs Black Electric Tesla). It's the survey form of discrete choice modelling. By randomising the options after each vote, you see which categories matter most and which options win inside each. It comes in several types, and if it turns out to be more than you need, there are simpler alternatives.
Conjoint analysis survey created for free on OpinionX ↗️
Q2. List length?
How many options are you trying to rank?
If you're ranking a short list of 3 to 10 options, continue to Q3. If your list has more than 10, jump ahead to Q4. Q4 covers why the number matters so much: long lists break the methods that ask people to see everything at once.
Q3. Equal or skewed input?
Should everyone have an equal say in picking one winner, or should stronger opinions carry more weight?
Since your list is 3 to 10 items, it's short enough for participants to evaluate the whole thing without being overwhelmed. That leaves two methods: order ranking and points allocation.
Order ranking asks participants to rank the entire list from most to least preferred, producing a clean set of priorities with no ties. It captures the relative position of preference: what comes first, second and third for each person. Because everyone's rankings count equally, order ranking is ideal when your goal is to choose a single winner.
Order Ranking survey created for free on OpinionX ↗️
Points allocation gives participants a pool of points to distribute freely across the list. It captures the relative weight of preference: not just which options people prefer, but how much they care about each one. Participants can emphasise what matters and skip what doesn't, which suits situations where you're picking multiple winners.
Points Allocation ranking survey created for free on OpinionX ↗️
Here’s an example to help demonstrate thHere's the difference in practice.
Say List A is testing podcast names. I want the single best option and I want everyone weighed equally, so order ranking does a better job of surfacing the most broadly appealing name.
List B is evaluating customer pain points to prioritise a product roadmap. Here I want the spiky problems that provoke strong opinions, even from a small segment shouting loudly, so points allocation is better: it lets people skip irrelevant issues and throw their weight behind what matters. Since I'll investigate the top 2 or 3 further, I care less about equal input and more about where opinions are loudest. That's the same logic behind needs-based segmentation and customer problem stack ranking.
Q4. Complex options?
How easy is it for participants to understand each option at a glance?
When your list has more than 10 options, showing them all on screen at once is overwhelming. Have you ever tried to rank 20 options on a phone? It isn't fun, and most surveys are answered on phones now. SurveyMonkey's platform data puts mobile at 58.2% of survey responses globally in 2024. That shift is why matrix grid questions have nearly vanished from online surveys, falling from 43% in 2015 to 23% in 2024, since a grid that needs horizontal scrolling is miserable on a phone.
Two methods collect ranked preferences for long lists without that problem.
Pairwise comparison shows your list as a series of head-to-head votes, an idea that goes back to paired-comparison work in psychophysics. Each option gets a score based on the percentage of pairs it wins. It handles very long lists that a drag-and-drop ranking can't.
Pairwise Comparison survey created for free on OpinionX ↗️
MaxDiff analysis shows 3 to 6 options at a time and asks participants to pick the best and worst in each set. Subtract the percentage of worsts from bests and you get the list ranked by preference.
MaxDiff Analysis survey created for free on OpinionX ↗️
Both handle long lists well, and both are flexible: you can reword the voting labels, for example changing MaxDiff's "best" and "worst" to "most important" and "least important". They aren't interchangeable, though. Three things decide which fits.
i. Complexity. Are your options short statements understood at a glance? Showing 3 to 6 long, complex statements on screen at once is overwhelming, which makes MaxDiff a poor fit. For longer statements I prefer pairwise comparison, which keeps things simple by showing two at a time.
ii. Population. How many people can complete your survey? If you're participant-constrained, MaxDiff gathers preference data faster than pairwise comparison, because it collects information on 3 to 6 options per vote instead of 2.
iii. Perspective. Sometimes I don't care about worst votes at all. If I'm ranking problem statements, I only want to know which problems frustrate people most, not which they consider low priority. MaxDiff balances best and worst into one score; pairwise comparison focuses solely on your ranking criterion. Choose accordingly.
What if you're ranking prices, not preferences?
One case the four questions don't cover: if the "options" you're ranking are price points, a preference ranking isn't the right instrument. Willingness to pay has its own methods. Van Westendorp finds a rough acceptable price range when you have no benchmark, and Gabor Granger finds the revenue-maximising point once you have a range to test. Reach for those rather than one of the five above when price is the thing you're measuring.
Frequently asked questions
What is the best survey method for ranking preferences? It depends on your list. For a short list of 3 to 10 where you want one winner, order ranking. For a short list where you want to weight strong opinions, points allocation. For a long list of simple options, MaxDiff analysis. For a long list of complex options, pairwise comparison. For options that span multiple categories, conjoint analysis.
What is the difference between order ranking and points allocation? Order ranking captures the position of preference, first to last, with everyone weighed equally, which is best for choosing a single winner. Points allocation captures the weight of preference by letting people spend a budget of points, which is best when stronger opinions should count for more and you're picking several winners.
When should you use MaxDiff instead of pairwise comparison? Use MaxDiff for short, simple options or when you have few participants, since it collects more data per vote. Use pairwise comparison for long or complex statements, or when you only care about the top of the ranking rather than the bottom.
Why not just use a rating scale? Because rating scales don't force a trade-off. People score most options highly, everything clusters between six and eight, and you can't tell what they'd actually prioritise. Ranking methods make people choose between options, which produces a result you can act on.
Which ranking method works best on mobile? Pairwise comparison and MaxDiff, because they show only a few options at a time. Long drag-and-drop rankings and matrix grids are hard to use on a phone, which is where most surveys are now answered.
Four questions, five methods. Single list or categories, short or long, equal or weighted input, simple or complex options. Answer those and you'll land on the right survey ranking method without agonising over the rest.
Want more like this? Over 42,000 researchers and product people get The Full-Stack Researcher in their inbox. Subscribe for the next one.
Every method here runs on OpinionX's free tier: $0, unlimited surveys, unlimited researcher seats, capped at 25 participants per survey, then $900 a year to lift the cap (full pricing).