Audit Reveals Methodological Flaws in Inference-Time Defense Evaluations for Multimodal LLMs

Jul 15, 2026

A recent audit of inference-time defenses for multimodal large language models (MLLMs) found that three benchmark branches failed provenance checks, restricting reliable comparative results to only two datasets. The study also discovered that a legacy keyword-based safety protocol incorrectly counted empty strings as safe, with raw model responses unavailable to reassess this effect. Contrary to earlier claims, the archive does not support widespread model refusal, with a pooled refusal rate of just 0.52%.

Why it matters: This work exposes critical methodological issues in evaluating MLLM safety defenses, emphasizing the importance of traceable audits and provenance standards for trustworthy comparisons.

Full story at: arXiv Cryptography and Security ↗