New Metrics Reveal Prompt Formatting Can Skew LLM Benchmark Results

Jul 14, 2026

A new arXiv study introduces the Format Sensitivity Index (FSI) and Parseability Sensitivity Index (PSI) to measure how prompt formatting affects large language model (LLM) benchmarking. Analyzing 140,000 generations across multiple models and tasks, the authors found that small changes in prompt wrappers can significantly alter model accuracy and leaderboard rankings. The research highlights that parseability is a strong predictor of accuracy and recommends reporting wrapper variance and compliance for more robust benchmarking.

Why it matters: These findings challenge the reliability of current LLM benchmarks and suggest new best practices for evaluating and deploying structured-output models.

Full story at: arXiv AI/ML ↗