Pew's Test of AI Survey Respondents Is a Warning for Anyone Doing Brand Research

Pew Research Center published a study on 30 September testing whether AI can stand in for people in surveys. It built digital twins of real American Trends Panel members from their demographics and earlier answers, then had Claude Opus 4.6 answer nearly 300 survey questions as each of them.

Across those questions, the AI estimates differed from the real human results by an average of 12 percentage points. The errors were not random. The AI overestimated one political approval figure (46 percent against an actual 34 percent), and awareness of data centres was 3 percent in the synthetic sample against 25 percent in reality. Errors were largest for Black respondents and Republicans, which Pew says suggests the model reinforces stereotypes.

Why this matters for creative work: "synthetic audiences" are being pitched for brand testing, naming and campaign research because they are quick and cheap. Pew's result is a useful reality check. A model gives you the average of what is written about a group, not what a person in that group thinks.

Pew also notes that it only tested AI as a replacement for respondents, not AI for coding or analysing responses, and says it has no plans to generate survey results with AI.

Our view: use AI to summarise real feedback, never to invent it. If a decision matters, talk to actual customers.