New Metrics Quantify Abstractiveness in Text Summarization

Jul 14, 2026

Researchers introduce three metrics—Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR)—to quantify how much a summary diverges from extractive copying of source text. Evaluated on 100 XSUM documents across four summarization models, these metrics effectively distinguish between extractive and abstractive approaches, with AR highlighting summaries that may require manual hallucination checks.

Why it matters: These metrics offer a principled alternative to traditional evaluation methods like ROUGE, enabling more nuanced assessment of summarization models and aiding in the detection of potential hallucinations.

Full story at: arXiv Computation and Language ↗