“Drag-and-drop ranking falls apart past about 10 options, and the big survey companies say so themselves. The two formats that do handle a long list, pairwise comparison and MaxDiff, are missing from every mainstream list ranking tool. Here’s why the standard rank order question breaks, and what to use instead.”

There's an endless number of survey tools online for ranking 5 options. But what should you use when you're trying to rank a list of 10 or more?

Or what if you've got 100+ options to rank? What then?

All the major survey companies make it pretty clear that you shouldn't use their products for this. SurveyMonkey advises limiting ranking choices to about five, because asking people to rank too many items against each other gets overwhelming. Qualtrics tells users to keep it between six and ten items, and adds that you shouldn't read too much into the middle rankings. Slido recommends keeping the list short, with a smiley face attached.

So they're telling you to shortlist the options yourself, which defeats the whole purpose of using a research tool in the first place.

Two survey formats are designed specifically for ranking long lists.

How To Rank Any Long List Of Options Survey Method

Why do survey companies tell you not to rank long lists?

Because the mental work required to drag-and-drop rank a list grows fast with every option you add.

To place an option properly, you have to compare it against the others you've already ranked. The number of possible comparisons in a list of n options is n(n-1)/2:

5 options: 10 comparisons
10 options: 45 comparisons
15 options: 105 comparisons
20 options: 190 comparisons

What looks like a simple task is a cognitive load problem that grows quadratically, and it's why drag-and-drop ranking produces straightlined junk data once the list gets long. Research on rank order questions has consistently found they get slower and more error-prone as options are added, which is the same central tendency territory that makes rating scales useless for ranking.

So in a way, the survey companies are right. You shouldn't use their tools to rank long lists of 10+ options.

Well, this begs the question.

Why aren't there question formats suited to longer lists?

There are.

Despite these companies raising billions in funding, they simply don't care about ranking research. Personally I find that pretty outrageous, because comparative ranking is one of the best ways to understand what people actually prioritise. But tools like SurveyMonkey aren't really built for research. They're online versions of traditional pen-and-paper questionnaires.

In 1927, the American psychologist L. L. Thurstone published A Law of Comparative Judgment, a paper describing a ranking approach where a list is broken into a series of head-to-head pair votes to measure people's values and attitudes.

Today we call it pairwise comparison, or Pair Rank for short.

The advantage of pair ranking

The best part is the flexibility.

Ranking a personal list? Complete all possible pair combinations and you get a full ranked list at the end. Got other people helping? Give each of them a smaller set of pairs to vote on, then aggregate everyone's results into the final ranking. That's a partial pairwise ranking, and it's how almost every group survey runs.

That flexibility stretches a long way. In 2021 I helped a researcher run a pair ranking survey with 800+ options and over 30,000 votes cast by participants. YouTuber Tom Scott pushed it further still, gathering 1.2 million votes across 7,188 options to rank everything in the world.

How does scoring work in a pair rank survey?

Whether your list has 1 voter or 10,000, the output is the same. Each option gets ranked from first to last using a 0 to 100 score based on the percentage of pair votes it won.

An option that appears in 5 pairs and wins 4 scores 80, because 4/5 = 80%. The formal model underneath is the Bradley-Terry model, published in 1952.

^ Results table for a Pair Ranking survey on OpinionX

A real example of ranking a long list

Hazel Byrne was completing her Master's in Global Challenges for Sustainability and had to pick a topic for her capstone project.

She was struggling to prioritise from a list of 33 options, so she ran a personal pair ranking survey to help her decide. The survey gave her every combination of those 33 options as head-to-head pairs and asked her to pick between Option A and Option B each time.

She ended up with a clear winner on a 100% win rate, and that's the topic she pursued.

Ranking masters project topics voting on capstone project ideas pair ranking example

Which list ranking tool has a pair ranking question?

No mainstream list ranking tool offers pairwise comparison. That includes Google Forms, Microsoft Forms, Qualtrics, SurveyMonkey, Typeform, Jotform, Slido and Mentimeter.

So we built OpinionX as a list ranking tool specifically, because we weren't happy with the formats the traditional survey platforms offered. Teams at Google, Amazon and Shopify use it to let their customers, colleagues and communities rank what matters most to them.

You can set exactly how many pairs each participant votes on, and include other question types alongside it like multiple choice and email collection.

pairwise ranking survey free tool — **left**: voting / **right**: results

Every question type and every analysis feature, including segmentation, works on the free tier. The cap is 25 participants per survey. Removing it costs $900 a year.

There's no reason not to give it a quick try. You can have your first pair ranking survey set up in under 5 minutes.

Create a FREE Pair Rank Survey 🚀

Need an even faster way to collect data?

MaxDiff analysis is very similar to pair ranking, except instead of showing two options at a time it shows 3 to 6 and asks participants to pick the best and worst from the set.

More options per vote means more information per vote, which makes MaxDiff a denser collection method than pairwise comparison. It's also known as best-worst scaling, though the two terms aren't quite interchangeable. The trade-off is a heavier task for the participant, so it suits shorter statements.

MaxDiff also sits on the OpinionX free tier, under the same 25-participant cap.

Frequently asked questions

How many options can a drag-and-drop ranking question handle?

About 6 to 10, and the major survey platforms say so in their own documentation. Past that, the number of mental comparisons grows quadratically, completion rates drop and the data quality goes with them.

What's the best way to rank 20+ options?

Pairwise comparison or MaxDiff. Pairwise shows two options at a time, which keeps each decision easy even with hundreds of options in the list. MaxDiff shows 3 to 6 per screen, which collects more data per vote when your participant pool is small.

How many comparisons are in a list of n options?

n(n-1)/2. Ten options produce 45 possible pairs, 20 produce 190, and 33 produce 528. Nobody has to vote on all of them: in a group survey each participant sees a sample, and the samples combine into the overall ranking.

Can one person rank a long list on their own?

Yes. Complete every possible pair combination and you get a full personal ranking, which is exactly what Hazel did with her 33 capstone topics. It only becomes impractical somewhere past 30 or 40 options, where the pair count climbs into the hundreds.

Which free list ranking tool offers pairwise ranking?

OpinionX is the one list ranking tool here that offers it free, with unlimited options and unlimited surveys, capped at 25 participants per survey. Google Forms, Microsoft Forms, Qualtrics, SurveyMonkey, Typeform, Jotform, Slido and Mentimeter don't offer the format at all.

Over 42,000 researchers and product people get one method breakdown like this each week in The Full-Stack Researcher.

A Simple Method For Ranking A Long List of Options