Interventional Grounding Audits: Black-Box Tests for LLM Chain-of-Thought Premise Dependency

Jul 16, 2026

Researchers introduce interventional grounding audits, a black-box method to test whether large language model (LLM) chain-of-thought reasoning genuinely depends on its stated premises. Applied to GPT-4o on the ProntoQA benchmark, the method achieves an F1 score of 0.806 for detecting proof-tree dependencies, significantly outperforming a self-consistency baseline. The study finds that 66% of correctly-solved problems contain reasoning steps that are insensitive to direct premise dependencies, highlighting cases of 'right answer, wrong reasoning.'

Why it matters: This work offers a scalable, black-box approach to identify when LLMs arrive at correct answers through flawed or spurious reasoning, addressing a key challenge for AI safety and reliability.

Full story at: arXiv AI/ML ↗