W2SPO: Weak-to-Strong Off-Policy RL via Auxiliary Branches Improves Reasoning in LLMs
A new method called W2SPO is introduced for reinforcement learning (RL) in large language models, addressing the challenge of limited exploration in reasoning tasks. W2SPO uses a weaker auxiliary model to inject short token segments into the target model's reasoning process, which helps diversify exploration and improve learning efficiency. On mathematical reasoning benchmarks at the 4B parameter scale, W2SPO outperforms post-trained baselines, raising Pass@1 from 62.3% to 64.2% and achieving a 3.55x training speedup compared to vanilla GRPO.
Why it matters: This approach offers a practical advance in RL for language models by overcoming exploration bottlenecks, leading to both faster training and improved reasoning performance.
Full story at: arXiv AI/ML ↗