Constant Sum Survey Method [Explanation & Real Examples]
What is a ‘Constant Sum’ survey question?
Constant Sum (also known as Points Allocation) is a survey method where respondents allocate a budget of points amongst a set of options according to their personal preferences. Here’s an example of a Constant Sum survey question:
Constant Sum is a great way to rank a set of options because it allows respondents to show the magnitude of their preferences, not just their ranked order. For example, we don’t just learn that Simon prefers apples to bananas, we see that he would give 9 of his 10 points to apples.
What’s an example of Constant Sum having real impact in a user research project?
I bet you didn’t know that Constant Sum played a important role in the early success of Google Docs!
Back when Google Docs was still in its beta phase, the team was working with pilot users to improve the product experience before public launch. As a beta product, it was still pretty rough around the edges, so the team was collecting user feedback about which aspects of the product needed improvement the most. They collected suggestions from users during informal interviews and testing sessions but the tallied results showed a large range of features users felt were needed, leaving the Google Docs team lost as to where they should focus their efforts — until they ran one simple experiment.
A Product Manager on the team asked beta users, “If you had $100 to spend on improving Google Docs, how would you spend it amongst this list of problems?”. The result was mind-blowingly clear — beta testers allocated 89% of all points on document formatting.
^ Google Docs, October 2006 (lacking a lot of formatting…)
They took this insight into user interviews and asked testers to explain their problems with document formatting in more detail. They realized that minor text changes could destroy an entire document’s format, requiring hours of work from the user to fix. This was a big enough problem that it often made beta testers rage quit and give up using Google Docs entirely.
The Google Docs team had discovered an important fact about user research — asking users to suggest problems is unreliable because it relies on problem recall at that exact moment, which people are notoriously bad at. Instead of counting a tally of feature/improvement requests, Constant Sum allowed the team to measure the magnitude of problem impact to their beta testers, surfacing the highest leverage problem to solve.
When is Constant Sum the right question to use in a survey?
A ranking format like Pairwise Comparison (ranking by voting on a series of head-to-head pairs of options) surfaces people’s implicit preferences by showing what matters to them without asking them to directly say so. Constant Sum, on the other hand, shows someone’s explicit preferences by asking them to indicate what matters to them and how much.
Implicit formats like Pairwise Comparison are a great way to understand people’s decision-making when their subconscious mind is in the driver’s seat, like ranking a list of potential names for your new idea, where it’s hard to rationally compare multiple options at once. Explicit formats like Constant Sum are superior for simulating rational decisions, like ranking which features in a paid subscription plan drive the most value for you as a customer.
What research questions could I answer with Constant Sum?
Here are three examples of survey questions that are suited to the Constant Sum format:
1. Customer Preferences
Like in the Google Docs example, Constant Sum is a great way to understand the magnitude of people’s preferences. If you’ve got a range of different options that you’re trying to decide between, like which features most entice your freemium users to upgrade to a paid subscription, then Constant Sum is often a more insightful ranking method than alternative comparison formats.
2. Value Association
Imagine you run a business that offers multiple services rolled into one — for example, you’re a graduation photographer who charges $50 to edit the photos, mail physical prints, and share digital copies for future use. You’re thinking about splitting these into separate products to increase your prices, but you don’t know which ones customers value most today. Giving participants $50 in imaginary points to divide based on how much they’d expect to pay for each separately will surface target customer’s perceived values.
3. Time Breakdowns
Casey Neistat had a famous YouTube video in 2015 called Fat and Lazy where he broke his average day up into 24 one-hour slots to show viewers what he spends his time on.
You can do this same exercise with your research participants using a Constant Sum question. The average American office worker spends 37 hours per week working, so give each participant 37 points and ask them to show what they spend their time on.
Which survey platforms offer a Constant Sum question type?
Even though it’s a very popular research method, many of the most popular survey tools do not offer a Constant Sum or Points Allocation question format, including Google Forms, Microsoft Forms, Typeform, SurveyMonkey, Slido, SurveyKing, or Jotform (surprisingly none of them!).
Alternative 1: OpinionX
All of the screenshots of Constant Sum questions throughout this post were taken from OpinionX — a research tool built specifically for creating ranking surveys. OpinionX is used by thousands of companies (including Google, Amazon and Microsoft) and all of OpinionX’s ranking formats, including its Constant Sum format “Points Rank”, are available on the free version.
↖ Constant Sum is called “Points Rank” on OpinionX
Free Version: The free version of OpinionX gives you unlimited surveys with unlimited questions and unlimited ranking options. All question types are available on the free version and you can have up to 50 participants per survey (no monthly/annual limits, just per survey).
Because OpinionX was built specifically for ranking, it comes with a bunch of purpose-built features for analyzing ranked data — for example, the segmentation feature lets you recalculate your results to see how different subgroups of your respondents voted compared to your overall results.
Premium Tiers: OpinionX’s “Pro” tier (100 participants per survey) costs $30/month and offers extra customizations like removing OpinionX branding or setting a minimum/maximum number of votes per participant. Advanced analysis like survey branching and results segmentation comes on the “Scale” tier (1,000 participants per survey) which cost $60/month. Check out the full OpinionX pricing breakdown for more info.
Alternative 2: Mentimeter
Mentimeter is an audience engagement tool built for classrooms that has a Constant Sum format called “100 Points”, which gives participants 100 points to allocate — but only in bundles of 10. It’s not possible to change it from 100 points or the 10-vote bundles. Also, Mentimeter’s free version has a limit of two questions per “presentation” and premium plans are charged per-year only.
Alternative 3: Polly
Polly is an app for running polls in Slack, Microsoft Teams and Zoom. It has an option for Constant Sum surveys, although they are limited to 10 options and a maximum budget of 100 points per participant. The free version allows for 25 monthly responses across all your polls/surveys.
Create A Free Constant Sum Survey
You can have your first Constant Sum survey up and running in less than 5 minutes on OpinionX. There’s no time limit to the free version and you don’t need to add your credit card details — the free tier genuinely is just free — so there’s no reason not to give it a try! Create a Constant Sum survey on OpinionX today.