Perceptive Humanoid Motion from Simulation
TL;DR HALO trains a Unitree G1 humanoid only on synthetic vision-language-motion data rendered in 1,000 procedurally generated rooms. Zero-shot, it sits, crouches, crawls, sidesteps through doors and pushes boxes in six unseen real homes, succeeding in 91% of 360 trials.
What they do
Humanoid policies that map pixels and language to joint commands need paired data, and teleoperation or wearable capture costs at least an hour of human time per hour of data. HALO generates that data in simulation instead: 300 hours of egocentric vision-language-motion episodes in 1,000 procedurally generated rooms, with no human in the loop. From it, a Unitree G1 learns six skills (walk to a named object, sit, sidestep through a half-open door, crouch or crawl under a table, push a box) and chains them on request, running fully onboard.
How it works
- Data. The Kimodo motion diffusion model generates whole-body motion from text prompts, waypoints and keyframes; geometric checks reject foot skating, floor penetration and joint-limit violations. Motions are placed in Infinigen rooms and rendered through a fisheye lens matching the robot’s camera, with appearance randomization.
- Orchestrator. An onboard VLM splits a free-form request into subtasks and decides when to advance, retry or insert a step.
- Motion generator. A frozen Qwen3.5-2B encoder and a flow-matching DiT map the RGB view and subtask to a 5 s whole-body plan, replanned at 3 Hz. The first 0.5 s of each plan is fixed to what the robot is already executing, so consecutive plans join smoothly.
- Motion tracker. A 50 Hz policy on depth and proprioception, distilled from per-skill PPO experts with privileged height maps, then fine-tuned with RL around a system-identified actuator model.
Results
- Single skills: 91% success over 6,000 simulated trials and 91% over 360 real trials in six homes never seen, scanned or mapped (from 85% for Sit to 100% for Crouch).
- Real-robot chains: Door→Go-To 100%, Crouch→Sit 60%, Crouch→Go-To→Push 50%.
- Sit ablations in simulation (full system 84%): 43% when trained in 1 room instead of 1,000; 24% without plan chunking and 66% with inference-time chunking instead of prefix conditioning; 19% with a regression decoder; 74% without depth in the tracker.
Limitations
Each new skill still needs a person to write its prompts and author its keyframes. The robot has no sense of contact, so it loses track of objects that leave the view or are occluded, and chained tasks often fail when a subtask starts with its target out of sight. The tracker follows the plan without knowing what it is for, and the orchestrator is simple.
Videos
Figures
Abstract
Humanoid robots have the potential to automate a wide range of physical tasks. Progress toward generalist humanoid autonomy, however, remains constrained by a data bottleneck: mapping pixels and free-form language to low-level joint commands requires paired supervision, and no corpus of such data exists at sufficiently large scale. Some approaches rely on human teleoperation and wearable capture devices, which require at least an hour of human time to produce an hour of data. We present HALO, a humanoid autonomy system powered by a pipeline that synthesizes vision-language-motion data across a thousand procedurally generated homes. The corpus scales with compute rather than human labor. HALO acquires a repertoire of physical skills, including sitting on chairs, navigating to named objects, stepping sideways through half-open doors, crouching and crawling beneath tables, and pushing boxes. It orchestrates these skills to solve composite, long-horizon tasks. Without ever seeing a real home during training, HALO transfers zero-shot to the physical world, achieving a 91% success rate across 360 trials in six unmapped real-world environments. See https://apple-aiml-research.github.io/ml-halo/ for code and videos.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.