Do Synthetic Users Work? What the Research Found, and Where It Breaks
Synthetic users work for one job: ranking and stress-testing options you've already written, in categories the model knows well. The best study so far got AI consumers to about 90% of real panels' own consistency on purchase intent. They don't work as a stand-in for real buyers. They're too agreeable and too shallow, and they only predict what people say. That's the verdict from the primary…
Synthetic users are a form of artificial intelligence designed to mimic human responses, specifically in the context of ranking and stress-testing options that a model has previously written. A study conducted by PyMC Labs and Colgate-Palmolive, published in arXiv in 2025, tested GPT-4o and Gemini 2.0 synthetic users on 57 personal care concept surveys, comparing their results to those of 9,300 real respondents.
The findings indicate that synthetic users can achieve about 90% of the consistency in purchase intent as real panels. However, synthetic users are not reliable stand-ins for actual buyers, as they tend to provide agreeable and superficial responses, and they only predict what people say rather than what they would actually do. The research highlights that the performance of synthetic users depends on various factors, including the source of the persona information, the method used to collect answers, and the voice of the synthetic user.
While synthetic users can provide valuable insights and help fill data gaps, they should not replace human-centered research, as they may not accurately capture nuanced consumer behavior and preferences.
Rewritten by urgent.news; may contain errors — check the original.
Source: dev.to