Study Finds Language Models Consistently Agree With Each Other More Than With Human Readers
A new arXiv preprint reports that large language models (LLMs) show significantly higher agreement with each other than with human readers when selecting important sentences from web documents. The study, which used naturally occurring human highlights as a reference, found that model-model agreement was more than double the agreement between human readers. This pattern held across multiple vendors, model sizes, and evaluation procedures, and was not explained by prompt wording or determinism. Smaller models showed agreement levels similar to humans, while larger models exhibited greater convergence.
Why it matters: The findings suggest that LLMs may produce less diverse outputs than human users, raising questions about the representativeness and diversity of AI-generated content.
Full story at: arXiv Computers and Society ↗