Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models

Jul 16, 2026

Researchers introduce DROPJ, a human-centered method for safe reinforcement learning in environments without a predefined reward function. The approach uses a world model learned from prior trajectories, human preferences over simulated trajectory segments, and justifications for those preferences to train a reward model. This reward model is then used with model predictive control for agent deployment. Experiments show that generating informative simulated trajectories reduces computational cost and can improve deployment performance, while safety justifications can enhance safety during deployment.

Why it matters: This work presents a novel approach for aligning AI agent behavior with human safety preferences in safety-critical environments where traditional reward functions are unavailable.

Full story at: arXiv AI/ML ↗