Reproducing LightMem: Retriever Choice and Token Budgets Drive Memory Management Performance
A reproduction study of the LightMem memory-management approach for long-term conversational agents finds that retriever choice can significantly affect answer accuracy, with performance varying from 58.1% to 75.5% depending on the retriever used. The study shows that Naive RAG, which retrieves directly from raw user turns, generally outperforms LightMem at matched retrieval depths, while LightMem only has an advantage under strict token constraints. The results suggest that LightMem provides a context-efficiency trade-off rather than a universal improvement over Naive RAG.
Why it matters: This challenges assumptions about the superiority of constructed memory in conversational AI and highlights the critical role of retriever selection and token budgets.
Full story at: arXiv Information Retrieval ↗