In an early pilot, GPT valued fluoride in toothpaste at $3.40, against $3.27 from a published study of shoppers. That's the kind of result behind the "just ask ChatGPT" advice product teams now hear before they pay for participants. Then the same researchers, James Brand, Ayelet Israeli and Donald Ngwe, ran the full studies on GPT and on people side by side (Harvard Business School working paper 23-062, revised April 2026). GPT put fluoride at $8.20 and $9.30, while people put it at $2.60 and $2.90, so the model came out roughly three times too high. GPT also predicted people would like pancake-flavoured toothpaste, and they don't.
So, should AI design your conjoint study? AI conjoint analysis is useful for drafting the design of a study from your own interview notes, and it's misleading for the answers, because synthetic respondents get rough averages and wrong customer segments. Segments are usually why you ran a conjoint study in the first place. This article covers why language models get preferences wrong, where they break worst, which stages of a study they can help with, and a rule for using them.
What did the toothpaste studies test?
Conjoint analysis is a survey method that shows people several versions of a product, each a different combination of features and a price, and asks them to pick the one they'd buy. From thousands of those choices, the analysis works out a score for every feature, which OpinionX's results call scores and textbooks call part-worth utilities. Divide a feature's score by the score for price and you get willingness to pay, meaning the extra amount a person would pay for that feature. Our guide on what conjoint analysis is and how to run one covers the method in full. Our guide to marginal willingness to pay shows how that dollar figure gets worked out.
Brand, Israeli and Ngwe gave GPT the same conjoint questions they gave shoppers and compared the two sets of results. The products were everyday ones, toothpaste and laptops, so the model had seen plenty of text about them. On a familiar feature like fluoride, the pilot result landed close to the human figure, but the main studies didn't repeat it.
Why does a language model get preferences wrong?
A language model predicts the most likely next words about a product, based on everything written about that product in its training text. That prediction sits close to the average opinion in the text, which is why GPT's answers on familiar features can land near the human figure. A fluoride question has decades of toothpaste reviews and adverts behind it.
New features are where the prediction breaks, because there's little or no text about them to average. In the Brand, Israeli and Ngwe studies, GPT gave the wrong sign to two new toothpaste flavours, cucumber and pancake, predicting people would pay more for them when people would pay less. GPT also valued a laptop with a built-in projector at more than three times what people did.
New features are usually what a product team is testing. Nobody runs a conjoint study to find out whether people like fluoride, because the market already answered that. Teams run conjoint to price a feature that doesn't exist yet, which is exactly where the model has the least to go on. Our guide on when to use conjoint analysis, and 10 times not to covers which product questions a conjoint study is built for.
The authors' own verdict on the untuned model is blunt: the baseline estimates are "too noisy and inconsistently signed to be acted upon directly". Inconsistently signed means the model sometimes gets the direction wrong, saying a feature adds value when it takes value away.
Why does AI conjoint analysis break on customer segments?
A customer segment is a group of participants you want to look at on its own, such as buyers on a lower income, or people in one job role, and segments are where synthetic respondents fail worst. Brand, Israeli and Ngwe split their laptop results by income. For buyers earning under $50,000 per year, people valued extra RAM at $17.80, while GPT put the same feature at minus $4.40, so the model said this group would pay to avoid more memory.
The same group shows a second problem. Every choice in the study included a "none of these" option, so a participant could reject all the laptops shown. GPT, answering as a lower-income buyer, chose "none of these" 95% of the time, against 49% for the people in that income group. A simulated market where almost nobody buys tells you nothing about which product version would sell.
The researchers then tried to fix this by fine-tuning GPT, meaning they trained it further on human answers from their own studies, labelled with each person's demographics. Fine-tuning moved the averages closer to the human figures, but it didn't recover the differences between groups. Men in the human data valued a MacBook at minus $50 and women at $249, while the fine-tuned model put them at $45 and $57, close together and far from both. The paper's conclusion is that the models "do not provide meaningful across-group estimates".
Ali Goli and Amandeep Singh at the University of Washington found the same problem from a different angle. They asked GPT-4 a classic patience question, whether to take a smaller reward now or a larger one later, in several languages (Marketing Science, 2024). Asked in Malay, GPT-4 chose the patient option 29% of the time, and asked in Russian, 4.9% of the time. A real person's preferences don't change with the language of the question, so the swing comes from the text GPT-4 learned in each language.
What does this do to a conjoint simulator?
A conjoint market simulator takes each participant's scores and predicts which product version each person would choose, so you can test plans and prices before launch. Our guide on how a conjoint market simulator works walks through it with examples. The simulator is useful because people differ, so one segment picks the cheap plan while another pays for the premium one, and the share of each group decides your revenue.
Synthetic part-worths (scores produced by a model instead of by people) flatten those differences, because every simulated buyer is a slight variation on the same average customer. The simulator then gives every segment roughly the same answer, so the share of preference by segment, meaning the percentage of each group predicted to pick each version, is invented. That's why a model can't do the segmentation part of a conjoint study for you, since the differences between groups are what the study is there to find.
OpinionX estimates scores for every individual participant using hierarchical Bayes, a statistical method that uses the whole group's answers to produce a reliable score for each person. Per-participant scores are what let you filter results by segment and compare groups, as our guide to participant-level data explains. The none of the above option on conjoint surveys then shows how often each group rejects every option, the same measure GPT got so badly wrong.
Want to see what segment-level scores look like when actual people answer? A demo walks through a conjoint results dashboard, with the segment filters and the simulator.
Where does AI help in a conjoint study?
AI helps with drafting the study design, as long as the draft starts from what your own customers told you. This is the Discovery Sandwich, a three-layer product discovery framework where qualitative interviews find the problems, a quantitative survey ranks those problems by importance, and follow-up interviews explain the result (our guide to the Discovery Sandwich). AI fits inside the top layer as a drafting tool, and it has no place in the middle layer, where people supply the answers. Unlike a general mixed methods research design, the Discovery Sandwich ties both layers to one product decision.
Draft attributes and levels from your own interviews
Feed a language model your interview transcripts and ask it for candidate attributes and levels. Attributes, which OpinionX's builder calls categories, are the features you test, such as price or storage, and levels, which the builder calls options, are the values each attribute can take, such as 128GB or 256GB. Starting from transcripts means the list comes from your customers. Without them, the model falls back on the internet's average idea of your product category. Our guide to thematic analysis shows how to code the transcripts first, so you can check the model's list against what people actually said.
Stress-test the levels before you launch
A language model is good at reading a draft design and finding problems in it. Ask it to flag levels that overlap, such as "fast delivery" and "next-day delivery" in the same attribute, and combinations that can't exist in real life, such as a free plan with a dedicated account manager. OpinionX's prohibited pairing setting stops those impossible combinations from appearing in the same product version.
Ask the model, too, whether your price levels cover the range buyers would actually pay, because a range that stops too low hides the price people would accept.
Goli and Singh say the same in their own paper: "we do not believe that GPT can be used to directly elicit preferences", while language models are useful for generating hypotheses to test with human participants. Our list of 15 assumption testing methods covers how to turn those hypotheses into tests. The rule for your team is short: AI writes the questions, and only humans answer them.
If you've got interview notes and want help turning them into attributes and levels that hold up, book a free setup call and a research expert will go through your draft design with you.
What can AI do at each stage of a conjoint study?
The table below takes each stage of a conjoint survey in order. AI is useful for the first three stages and misleading for the last three, which is the reverse of where most AI conjoint analysis pitches put the value, since they sell synthetic respondents for the answering stage.
| Stage | What AI is good for | What AI gets wrong | Evidence |
|---|---|---|---|
| 🟢 Choosing attributes | Pulling candidate features out of your own interview transcripts | Falls back on the category average if you don't give it your own notes | Goli and Singh (2024): useful for generating hypotheses |
| 🟢 Writing levels | Suggesting realistic values and wording for each attribute | Can suggest price levels that don't match what your buyers pay | Check every price level against your own sales data |
| 🟢 Checking the design | Flagging overlapping levels and impossible combinations | Misses product rules it doesn't know about | Review step only; the design decision stays with you |
| 🔴 Answering the survey | Rough averages on familiar features | Overvalues new features, gets some signs wrong | Fluoride $8.20 and $9.30 vs $2.60 and $2.90; pancake and cucumber signs wrong (Brand et al., 2026) |
| 🔴 Splitting by segment | Nothing you can act on | Flattens differences between groups, even after fine-tuning | Under-$50,000 RAM: $17.80 for people vs minus $4.40 for GPT (Brand et al., 2026) |
| 🔴 Running the simulator | Nothing you can act on | Share of preference by segment is invented, and rejection rates are wrong | "None of these" chosen 95% of the time by GPT vs 49% by people (Brand et al., 2026) |
The last three rows are the stages that turn a conjoint study into a pricing or roadmap decision. The Harvard Business School AI Institute's summary of the research (August 2025) frames language models as an addition to research with people, never a replacement for it. If finding real participants is the cost you're trying to avoid, our guide on how to find free survey participants covers three ways to recruit them. Our guide to survey quotas shows how to make sure each segment you plan to compare gets enough people. Our article on AI in UX research covers the wider change in how researchers use AI.
Frequently asked questions
Can ChatGPT replace survey respondents in a conjoint study? ChatGPT can't replace survey respondents in a conjoint study. In Harvard Business School research by Brand, Israeli and Ngwe (revised 2026), GPT valued fluoride in toothpaste at about three times the human figure and got the direction wrong on two new flavours.
Are synthetic respondents accurate for willingness to pay? Synthetic respondents give rough willingness-to-pay figures for familiar features and unreliable ones for new features. GPT valued a laptop's built-in projector at more than three times what people did (Brand, Israeli and Ngwe, 2026), and new features are usually what a conjoint study tests.
Can AI choose the attributes and levels for a conjoint survey? AI can draft attributes and levels for a conjoint survey when you give it your own customer interview transcripts. Without them, a language model falls back on the general opinion in its training text. Check the draft against your interviews and test the final design with real participants.
Why do LLMs get customer segments wrong? Large language models predict the average opinion in their training text, so they flatten differences between customer groups. For buyers earning under $50,000, people valued extra laptop RAM at $17.80, while GPT said minus $4.40, and fine-tuning didn't recover the group differences (Brand, Israeli and Ngwe, 2026).
How should you use AI in conjoint analysis? Use AI in conjoint analysis to draft attributes and levels from your own interviews and to check the design for overlapping levels and impossible combinations. Collect the answers from real participants, because synthetic respondents get segment results and simulator shares wrong.
To understand the full method before you design one, our guide on how to calculate conjoint analysis results walks through the eight steps from raw choices to scores. If your study needs to compare customer groups, our needs-based segmentation guide shows how to define segments by what people need rather than by demographics. And if you're testing a product that doesn't exist yet, our guide to product concept testing covers which study to run first.
OpinionX's free plan includes conjoint analysis and every other method, capped at 25 participants per survey, so you can check a design with real people before you pay (see plans).
