New Method for Estimating Rare Events in Language Models with Proper Evaluation
Researchers introduce GA-AMLS, a rare-event Monte Carlo method that operates in the continuous activation space of language models to estimate extremely low probabilities, addressing issues like zero-estimate collapse and systematic bias found in previous approaches. They also propose the Shifted-Power Bregman Loss, a proper scoring rule that remains finite for zero estimates and allows for tunable asymmetry between underestimation and overestimation penalties. Experiments on small transformer models show that GA-AMLS achieves the lowest loss under symmetric evaluation, while methods with overestimation bias perform better under asymmetric penalties. The work demonstrates that estimator choice should be matched to deployment context and establishes activation space as a tractable domain for rare-event estimation in language models.
Why it matters: This work advances the robustness and reliability of rare-event risk estimation in language models, which is crucial for ensuring safety in large-scale and adversarial applications.
Full story at: arXiv Machine Learning ↗