Do LLMs Ask the Right Questions? Evaluating GPT-Generated Surveys as Instruments for Measuring Social Attitudes
A new preprint presents a controlled comparison between GPT-generated surveys and established, human-designed surveys across three social domains: climate change, immigration, and diversity, equity, and inclusion (DEI). The study finds that GPT-generated surveys capture the same dominant attitudinal divisions as human-designed instruments, though they differ in the resolution of belief structures and group separation. The authors conclude that LLM-generated surveys are suitable for exploratory and large-scale analyses, and can complement expert-designed instruments.
Why it matters: This work provides empirical evidence on the potential and limitations of using LLMs to automate survey generation for social attitude research.
Full story at: arXiv Computers and Society ↗