Data-Efficient Adaptation of LLMs via Attention Head Reweighting

Jul 16, 2026

Researchers introduce Attention Head Reweighting (AHR), a method for adapting large language models (LLMs) to new text-classification tasks by learning a single scalar per attention head. AHR achieves better performance than LoRA on few-shot tasks while requiring 200-1000 times fewer trainable parameters, modifying only about 0.0001% of the model. The approach also provides interpretable weights that help analyze which attention heads contribute to in-context learning.

Why it matters: AHR offers a highly parameter- and data-efficient way to adapt LLMs, which is valuable for applications with limited labeled data and enhances interpretability of model behavior.

Full story at: arXiv Machine Learning ↗