LLMs Outperform Traditional ML in Open-Ended Survey Analysis but Face Consistency and Explainability Issues

Jul 15, 2026

A new preprint compares large language models (LLMs) such as GPT, Twitter-roBERTa, and LLaMA to traditional machine learning methods for analyzing open-ended survey responses. The study finds that LLMs achieve higher classification accuracy, especially in sentiment and thematic analysis, but exhibit significant variation in consistency and the explicitness of their reasoning. These results highlight important trade-offs between predictive performance and interpretability in large-scale qualitative research.

Why it matters: The study offers practical insights for researchers seeking to balance automation with interpretive rigor when applying LLMs to qualitative data analysis.

Full story at: arXiv Computers and Society ↗