ROBIN: Localizing and Repairing Bias in Transformer Attention Heads

Jul 15, 2026

A new method called ROBIN introduces white-box, head-level fairness debugging for transformer models by ranking attention heads based on their sensitivity to fairness probes and removing a small bias subspace from selected head outputs. In a four-model pilot study, ROBIN reduced the WinoBias gap while preserving language-modeling quality better than whole-head zeroing. The results indicate that both the selection of attention heads and the method of modification are important for effective bias repair.

Why it matters: This work demonstrates a targeted inference-time approach for mitigating bias in transformer models, offering a more precise alternative to retraining or coarse interventions.

Full story at: arXiv Software Engineering ↗