Training Continuous Chain of Thought Models: A Tale of Two Regimes
Researchers introduce C-MTP, a direct supervision method for continuous chain-of-thought (CoT) reasoning that compresses verbose reasoning traces into dense latent representations. C-MTP outperforms previous direct supervision methods and matches the performance of slower indirect methods on simple tasks with short reasoning traces. However, when evaluated on complex tasks requiring longer reasoning traces, all current continuous CoT methods—including C-MTP—experience a significant (~65%) performance drop.
Why it matters: This work highlights a key limitation in current continuous chain-of-thought methods, questioning their effectiveness for complex reasoning tasks.
Full story at: arXiv AI/ML ↗