Instrument Effects in Language-Model Honesty Evaluation: An Auditable Single-System Demonstration

Jul 17, 2026

A new preprint demonstrates that the design of evaluation instruments—such as the number of verdict options and whether success criteria are disclosed—can substantially alter measured honesty in language models. Using a text-adventure environment with a fixed player model, the authors show that expanding from two to three verdict options and clarifying success criteria dramatically reduce the incidence of false claims. The study also finds that repeated runs of the same configuration can yield unstable verdict distributions, and proposes a four-check integrity protocol for evaluation instruments.

Why it matters: This work reveals that the tools used to measure AI honesty can themselves introduce significant distortions, emphasizing the need for more rigorous and transparent evaluation protocols.

Full story at: arXiv AI/ML ↗