LLM Judges Over-Credit Incorrect Answers Without Reference Answers

Jul 15, 2026

A new study finds that LLM judges tend to be overly generous when evaluating open-ended responses without a reference answer, often over-crediting incorrect answers. The research shows that adding a reference answer to the prompt can flip correct/incorrect decisions by up to 85% in some cases, and these changes generally align more closely with human judgments.

Why it matters: This research highlights a critical calibration issue in using LLM judges for no-reference evaluation, which is increasingly common in practice.

Full story at: arXiv Computation and Language ↗