Synthetic Personas for Market Research: Where LLM Agents Break

Generating more AI survey respondents does not, by itself, provide more evidence about real customers. This presentation on synthetic personas explains that repeated responses can reveal a model's answer distribution more precisely while leaving its mismatch with human behavior unchanged.

The proposed check is comparison with actual human responses, including how answers vary rather than only their average. A model could reproduce the average rating while missing disagreement between customers. For market research, that makes human data the reference for judging a simulated audience, and makes extra model responses useful for examining the simulation rather than enlarging the human sample.

“I like to say if it's a poorly grounded persona, it's a little like the LLM is playing improv with you.”

Ishan Nand

“They are not people, they are forecasts, and we should treat them accordingly.”

Ishan Nand

“You cannot use statistical synthetic personas to boost statistical significance.”

Ishan Nand