Shared 'Countdown Subcircuit' Enables LLMs to Track Remaining Tokens Across Diverse Tasks

Jul 15, 2026

Researchers have identified a general 'countdown subcircuit' in Llama-3.1-70B-Instruct that compares the current position to a goal length, allowing the model to estimate remaining tokens. This subcircuit uses an identical motif previously found in a frontier LLM on a different task, indicating shared mechanisms across models. Unsupervised probing revealed that the subcircuit is employed in various natural language tasks, including those where the goal length must be inferred from context.

Why it matters: This finding advances mechanistic interpretability by revealing a shared neural mechanism for token counting across tasks and models, potentially improving our understanding and control of LLM behavior.

Full story at: arXiv Computation and Language ↗