Controlled Reformulation Testing Reveals Logical Consistency Gaps in Large Language Models

Jul 17, 2026

A new benchmark, CRTBench, evaluates large language models (LLMs) for logical consistency across controlled reformulations such as contrapositive and double negation. The study finds that while models like GPT-5.4-mini achieve high base accuracy (98.9%), their consistency across logically equivalent reformulations is much lower (60.3%), compared to reasoning-optimized models like o4-mini (96.9%). The results demonstrate that LLMs often fail on logically nontrivial transformations, highlighting a significant gap between accuracy and true logical reasoning.

Why it matters: This work exposes a critical limitation in current LLM evaluation, showing that high accuracy does not guarantee logical consistency, and calls for more robust benchmarks.

Full story at: arXiv Computation and Language ↗