S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF
Researchers introduce S2T-RLHF, a sentence-to-token reward decomposition framework designed to improve training stability in reinforcement learning from human feedback (RLHF). By assigning preference rewards at the sentence level before refining them at the token level, S2T-RLHF addresses instability caused by noisy, overly fine-grained token-level credit assignment. Experiments demonstrate that this approach enhances stability and robustness while maintaining competitive alignment performance.
Why it matters: This work proposes a novel granularity-aware credit assignment method that could make preference-based RLHF training more stable and reliable for large language models.
Full story at: arXiv AI/ML ↗