Short-Answer VQA Scores Confound Semantic Correctness with Surface-Form Match, Study Finds

Jul 14, 2026

A new study finds that short-answer Visual Question Answering (VQA) benchmarks often conflate semantic correctness with surface-form matching, leading to many semantically correct answers being penalized for not matching the expected format. By auditing over 37,000 official errors across six models and benchmarks with a human-validated semantic judge, the authors show that up to half of errors on text-rich benchmarks are due to this issue. Extractive and multi-span answers are especially sensitive to evaluator criteria, and even benign prompt rewrites can flip item-level correctness.

Why it matters: This challenges the interpretability of widely used VQA benchmarks and suggests that official scores should be supplemented with semantic audits and answer-type diagnostics.

Full story at: arXiv Computer Vision ↗