Contrastive Policy Optimization Improves RL with Verifiable Rewards

Jul 17, 2026

Researchers introduce Contrastive Policy Optimization (CPO), a method that leverages token-level contrastive disagreement between reference-guided and vanilla generation distributions for correctness-aware advantage shaping in reinforcement learning with verifiable rewards. The study presents both theoretical and empirical evidence that CPO provides a more reliable correctness signal than traditional entropy-based methods. Experiments show that CPO outperforms entropy-based approaches on both in-domain and out-of-domain benchmarks while maintaining strong generalization.

Why it matters: This work offers a novel approach to improving the reliability and effectiveness of reinforcement learning from verifiable rewards, potentially advancing the field's practical capabilities.

Full story at: arXiv Machine Learning ↗