Structured Output Collapses Answer Diversity Across 44 Language Models
A new preprint reports that when 44 language models are prompted to reply in structured formats like JSON, their answer diversity drops significantly compared to plain chat. The modal answer rate increases from 41% to 64%, and the number of distinct answers falls from 52 to 36. This effect is specific to formats such as JSON and XML, and is caused by the request for structured output itself, not by decoder constraints.
Why it matters: The findings suggest that the structured output formats widely used for software integration lead to less diverse model responses than the chat interfaces typically used for evaluation, potentially impacting downstream applications and model comparisons.
Full story at: arXiv Computation and Language ↗