DevicesWorld: Benchmarking Cross-Device Agents in Heterogeneous Environments

Jul 16, 2026

DevicesWorld is a new benchmark designed to evaluate LLM-based agents on tasks that require collaboration across mobile, desktop, and IoT devices. The benchmark features 6,140 tasks and a unified evaluation framework, revealing that current leading agents achieve only a 12.5% success rate. Analysis of agent failures highlights common issues such as difficulties in information acquisition and confusion between source and output devices.

Why it matters: DevicesWorld fills a critical gap by enabling systematic evaluation of agents' abilities to operate across heterogeneous device environments, which is essential for real-world applications.

Full story at: arXiv Multiagent Systems ↗