Branching Policy Optimization: Sandbox-Native RL for LLM Agents
Researchers introduce Branching Policy Optimization (BPO), a reinforcement learning algorithm designed for large language model (LLM) agents operating in deterministic, snapshottable sandboxes. BPO leverages the ability to fork alternative actions at high-entropy decision points, sharing rollout prefixes to reduce variance and compute unbiased, lower-variance advantage estimates from sibling returns. Experiments on WebShop, ALFWorld, and SWE-bench Verified show that BPO improves success rates by 3.6–6.1 absolute points over GRPO and RLOO at matched compute, and achieves similar performance to the best baseline with 38% fewer policy updates.
Why it matters: BPO demonstrates a novel and more efficient approach to RL for LLM agents by exploiting sandbox determinism, leading to improved sample efficiency and training stability.
Full story at: arXiv Machine Learning ↗