PReM: Learning What to Preserve and When to Refresh for Context Compression

Jul 17, 2026

PReM is a context-compression framework for language models that maintains long context as internal layer-wise key-value (KV) memory, learning dynamically what information to preserve and when to refresh it. The approach introduces a dedicated memory layer and a special memory token, trained using Phase-Separated Refresh Training to align memory selection with generation needs. Experiments on 32K-token contexts demonstrate that PReM outperforms strong baselines at 16x and 32x compression, achieving a favorable balance between answer quality and inference efficiency.

Why it matters: This work offers a novel approach to efficient long-context inference by enabling dynamic adaptation of compressed context, potentially improving both quality and efficiency in large language models.

Full story at: arXiv Computation and Language ↗