Metadata-Free Meta-Reweighted DPO Improves Alignment Under Noisy Preferences

Jul 14, 2026

Researchers introduce a bilevel optimization framework for Direct Preference Optimization (DPO) that can recover clean-data performance even when preference labels are noisy. Their approach uses meta-learning without requiring metadata, leveraging central-difference approximation and LoRA for scalable training. Experiments on TL;DR summarization and Anthropic HH dialogue tasks demonstrate improved performance over standard DPO baselines across various noise rates.

Why it matters: This work addresses a key limitation of DPO by enabling robust alignment of language models in the presence of noisy preference data, which is common in real-world applications.

Full story at: arXiv Machine Learning ↗