Open-AoE: Open Egocentric Manipulation Dataset and Toolchain for Embodied Learning

Jul 17, 2026

Open-AoE is an open, community-oriented dataset and toolchain for egocentric manipulation, featuring approximately 2,000 hours of video collected by over 500 contributors using more than 400 smartphones. The dataset includes structured annotations such as text labels, MANO-based hand poses, camera trajectories, and temporally localized atomic actions. It also provides a full pipeline for data processing, including temporal action segmentation, semantic annotation, hand and camera trajectory reconstruction, as well as downstream tools for visualization, cross-embodiment retargeting, and model training. This resource is designed to facilitate embodied AI research and human-to-robot transfer by lowering barriers to data contribution and reuse.

Why it matters: Open-AoE provides a large-scale, practical infrastructure for embodied AI research, potentially accelerating advances in manipulation tasks and world modeling.

Full story at: arXiv Robotics ↗