LLMs Struggle to Track Source of Information Over Multi-Turn Conversations
A new arXiv preprint investigates whether large language models (LLMs) can reliably distinguish between their own outputs and user inputs—a cognitive skill known as reality monitoring. The study finds that while LLMs perform well at this task when memory demands are low, their accuracy drops and sometimes reverses when conversation history is extended, leading to confusion about the source of information. The research also uncovers dissociations between confidence and correctness, and between internal and external attributions, that are not captured by standard benchmarks.
Why it matters: This highlights a potential risk for AI systems deployed in autonomous, multi-turn settings, where misattributing the source of information could lead to compounding errors or hallucinations.
Full story at: arXiv Computers and Society ↗