Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents

Jul 15, 2026

A new preprint introduces a framework for co-evolving evaluation metrics alongside skills in self-improving large language model (LLM) agents, addressing the common but often unexamined assumption that reliable evaluation metrics are already available. The proposed Double Ratchet system demonstrates that co-evolved metrics can retain 88–110% of the performance improvement ('lift') achieved by ground-truth-driven skill loops across tasks such as code generation, text-to-SQL, and report generation. The approach incorporates safety mechanisms like anchor discipline and external audits to mitigate metric gaming and ensure transparency.

Why it matters: This work offers a practical and auditable method for evolving evaluation metrics in autonomous AI systems, potentially making self-improving agents more robust and trustworthy in real-world applications where ground-truth metrics are unavailable.

Full story at: arXiv AI/ML ↗