EgoSwarm: Learning Decentralized Robot Swarm Intelligence from Human Swarm Demonstrations
- arXiv
- Ask alphaXiv
- Project page
- Imitation learning and data collection
- Humanoids and mobile manipulation
TL;DR EgoSwarm records people collaborating with head-worn sensors (about 50 hours) and turns the recordings into training data for one shared VLA policy that each robot runs on board, exchanging only short messages with its neighbors. On three real three-robot tasks it reaches 72.2% mean success, against 22.2% when trained without the human data.
What they do
Multi-robot collaboration (handovers, carrying a shared object) needs data on how agents depend on each other, which single-robot demonstrations lack and multi-robot teleoperation makes expensive. EgoSwarm instead records groups of people doing the tasks with head-worn sensors, then trains a decentralized policy that each robot runs using only its own observations, history and short messages from its neighbors.
How it works
- EgoSwarm Capture. Each person wears a sensing head with four RGB cameras, two infrared cameras, an IMU and a UWB tag; about 50 hours were collected across 30 scenario categories. The robots (mobile bases with two 7-DoF arms) carry the same head.
- Swarm Interaction Packets. Recordings are time-aligned, SLAM maps of the participants are registered within each interaction window, and events such as waiting and handovers are annotated. Fields are split into what a robot can observe online and labels used only offline.
- SwarmVLA. Starting from π0.5, each robot encodes relative geometry, shared-object and contact state, uncertainty and link quality into tokens; a quality-gated attention layer weights neighbor messages and a flow-matching expert outputs action chunks.
- Training. A teacher that sees the reconstructed group information is distilled into the shared local student, with simulated message loss and delay.
Results
- Over 30 real trials per task, EgoSwarm reaches 80.0% (garment handover), 73.3% (box transport) and 63.3% (wire harness), a 72.2% mean, against 42.2% for independent VLAs, 50.0% for VLAs with plain message passing and 58.9% for a centralized VLA with global observations.
- Without human swarm data the mean is 22.2%; with half of it, 53.3%. Removing the interaction tokens drops it to 38.9%.
- With 30% packet loss and 100 ms extra delay the mean stays at 66.7%.
- Each robot runs the policy on board in 14.1 ms per inference and keeps closed-loop control at 30 Hz.
Limitations
The system relies on visual, inertial and UWB signals without touch or force sensing, reconstructs interactions only within well-observed local windows, and is evaluated mainly with three robots. Training also uses robot adaptation data and task-specific joint teleoperation.
Videos
Figures




Abstract
Advancing from individual embodied intelligence to swarm embodied intelligence is a key step toward scalable, general-purpose embodied intelligence. Complex tasks often require multiple agents to divide responsibilities, coordinate handovers, and manipulate shared objects collaboratively. However, existing robot learning methods rely primarily on individual demonstrations, which provide limited supervision for modeling dependencies among agent actions, while collecting multi-robot demonstrations at scale remains difficult. Human egocentric demonstrations offer a scalable alternative, yet distributed observations from multiple participants cannot directly supervise decentralized policies. We introduce EgoSwarm, the first robot learning framework for learning decentralized multi-robot collaboration from human swarm demonstrations. EgoSwarm collects more than 50 hours of collaborative experience using a robot-free multi-person egocentric system. We introduce the Swarm Interaction Packet (SIP) and Swarm Interaction Token (SIT) to recover cross-agent interaction relations from distributed observations and transform this experience into local supervision consistent with deployment-time information constraints. We further propose SwarmVLA, which transfers knowledge of swarm behavior to decentralized policies conditioned on local observations, local history, and limited neighbor messages. SwarmVLA achieves a mean success rate of 72.2% across three categories of multi-robot collaborative tasks. Project page:https://xitong-c.github.io/EgoSwarm_Page/.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
