Forced Choice Questions: Use Cases, Survey Examples, Ranking
What is a Forced Choice Question?
Forced Choice Questions are when participants are not given any “nonresponse” options like Skip or Not Applicable. They are commonly used as forced choice rating scales (eg. 1-5 star rating) or forced choice ranking questions (eg. forced pairwise comparison).
When is Forced Choice Question used?
To understand when to use forced choice question, you first need to know when it shouldn’t be used. The main risk of forced choice surveys is that participants can be forced into selecting options that are not relevant/applicable to them because they had no option to exclude them. To avoid this risk, forced questions should only be used in surveys that meet one of these conditions (particularly for ranking questions):
Participants are answering opinion-based options that everyone can consider, such as voting on new team names, picking a group activity, choosing a party theme, or ranking hypothetical statements.
Participants all share the same perspective — ie. they have been specifically recruited/screened based on specific criteria — which guarantees that the ranking options apply to everyone.
If your survey does not meet these criteria, you should probably include a ‘Skip’ or ‘Not Applicable’ option to prevent participants from unwillingly submitting junk data.
What are the benefits of Forced Choice Questions?
People run surveys to gather actionable responses. When participants select options like ‘Skip’, ‘Not Applicable’ or ‘Indifferent’, it gives you less ‘opinionated data’ that can be used to inform decisions. Forced choice helps to fill in those gaps by making responses mandatory.
Forced ranking has an additional benefit which is unique compared to other types of survey questions. Ranking helps us understand how people make decisions by asking them to compare a set of possible choices. If we allow participants to exclude some choices or skip certain votes, it gives us an incomplete picture of their decision-making rationale. Therefore, for some ranking projects, forced choice questions give us a more complete picture of the trade-offs that people are willing to make in reality.
What are the risks of using Forced Choice Questions in your survey?
1. Nonresponse Bias
Forcing participants to rank options that are not applicable, relevant or understandable to them will result in them submitting junk data in your survey (‘nonresponse bias’). Avoid this by only using forced questions when your response options meets the criteria listed under the previous section When is a Forced Choice Question used?.
2. Acquiescence Bias
Forced choice survey questions have a higher rate of acquiescence bias — the tendency for participants to agree with research statements even if they are not a true reflection of their own opinion. However, ranking surveys do not have the same risk of acquiescence bias because they are comparative rather than absolute (eg. a ‘yes/no’ question), so this is less of a concern for ranking research.
3. Poor Survey Design
One of the most common reasons for participants to submit junk data is poor survey design. When a survey is designed poorly, forced choice question can exacerbate the problem by removing participants’ ability to skip a question rather than submit junk data out of impatience, tiredness or frustration.
One of the most common ways I see this in surveys is when participants are asked to drag-and-drop a long list of 10+ options into their ranked order of preference. If you’d like to use forced choice for ranking a long list of 10+ options, you should use a Pair Ranking format instead. Pair Ranking (also called Pairwise Comparison) asks participants to vote on a series of head-to-head pairs of options rather than trying to rank all options at the same time.
What free tools can I use to build a forced choice ranking survey?
While some of the tools below are able to offer forced choice rating scales, my focus particularly is on forced choice ranking questions. I previously wrote a deep-dive comparison of the most popular survey tools for ranking research — this post uses that same list of tools but focuses only on their ability to be used for creating forced choice ranking surveys:
⬛ Google Forms: No ranking question types available at all.
🟥 Slido: Ranking question type available but no forced choice option.
🟥 Qualtrics: Ranking question type available but no forced choice option.
🟥 Mentimeter: Ranking question type available but no forced choice option.
🔶 Typeform: Allows for Rank Order questions where you can enable the “Required” toggle to make it forced choice ranking, however, Typeform only allows for 10 participants per month on its free tier. No ‘Pair Rank’ format available.
🔶 SurveyMonkey: Allows for Rank Order questions where you can enable the “Required” toggle to make it forced choice ranking, however, SurveyMonkey only allows ranking questions for premium customers. No ‘Pair Rank’ format available.
🟢 OpinionX: Comes with multiple ranking formats (Pair Rank, Order Rank, Agreement Rank) and they each come with an optional “Required” toggle for enabling forced choice ranking. This is all available on the free tier of OpinionX (an online tool for stack ranking people’s priorities).
Setting up an Order Rank question on an OpinionX stack ranking survey so that it is Forced Choice.
OpinionX is used by thousands of teams around the world to better understand people’s priorities, giving product teams from some of the biggest tech companies the objective data they need for prioritizing what to focus on next. Best part is that it’s free and super easy to use — build your first stack ranking survey in minutes today!