Kepler-Encoder-v0.1: A Multimodal Embedding Model That Fuses Robot State and Vision

Jul 16, 2026

Researchers present Kepler-Encoder-v0.1, a multimodal encoder that fuses vision, proprioception, and force/torque data into a shared latent space using cross-attention and self-supervised learning. At inference, only vision is used, yet the latent representation recovers force and end-effector state information better than vision-only baselines, particularly in scenarios where visual input is limited. The encoder generalizes across four different robots, and its latent can serve as a training-free invalid-state monitor.

Why it matters: This work demonstrates that incorporating robot state during training enables vision-only representations to capture information about force and contact, potentially improving robot perception and safety without extra sensors at runtime.

Full story at: arXiv Robotics ↗