AI21 Labs finds LLM judge bias in coding agent benchmarks, warns of 'gold-like' answer pitfalls

Jul 11, 2026

AI21 Labs discovered that their LLM judge, used to select the best output from parallel agent runs in their Maestro coding agent, was performing suspiciously well. After ruling out contamination, they found the bias persisted on a clean dataset, indicating a deeper issue with benchmark evaluation.

Why it matters: This finding highlights a critical flaw in using LLMs as judges for coding agent benchmarks, potentially inflating performance metrics and misleading progress in AI development.

Full story at: AI21 Labs ↗