RLearner-LLM: Hybrid Direct Preference Optimization Balances Logical Grounding and Fluency
A new method called RLearner-LLM with Hybrid-DPO is proposed, combining a DeBERTa-v3 NLI signal and a verifier LLM score to address verbosity bias in standard Direct Preference Optimization (DPO) for language models. Evaluated across five academic domains and three model architectures, the approach achieves up to 6x improvement in NLI entailment over supervised fine-tuning (SFT), with consistent gains in answer coverage and effective scaling to compact models like Gemma 4 E4B-it. The method also demonstrates faster inference and mitigates the alignment tax associated with preference optimization.
Why it matters: This work demonstrates a significant advance in preference optimization for knowledge-intensive language generation by showing that hybrid logic-aware signals can improve logical correctness without sacrificing fluency, highlighting the limitations of current LLM-as-a-judge metrics.
Full story at: arXiv Computation and Language ↗