Humanoid Locomotion as Next Token Prediction

Ilija Radosavovic, Bike Zhang, Baifeng Shi, Jathushan Rajasegaran, Sarthak Kamat, Trevor Darrell, Koushil Sreenath, Jitendra Malik

Seminal February 2024

Digit walks along a downtown San Francisco sidewalk as a cyclist passes.

TL;DR Train a small causal transformer to predict the next observation and action of humanoid walking trajectories, masking actions where the data has none (motion capture, YouTube videos). Trained offline on 27 hours of walking data, it walks a full-sized Digit humanoid around San Francisco zero-shot and tracks commands better than the RL policy that generated its labeled data.

What they do

They treat humanoid control like language modeling: a causal transformer learns to predict the next token of sensorimotor trajectories. Because it predicts observations as well as actions, it can also learn from trajectories without actions, such as human motion capture and YouTube videos retargeted to the Agility Robotics Digit humanoid.

How it works

Data. 10k action-labeled 10 s trajectories from a prior RL policy; 20k observation-only trajectories from Agility’s model-based controller (it outputs torques, not the joint-position actions used here); about 1k KIT MoCap clips and PHALP-reconstructed YouTube humans, both retargeted to Digit by inverse kinematics.
Model. A vanilla transformer (4 layers, hidden size 192, 16-step context) regresses the next token with an MSE loss; observation and action at each step share one token by default. Missing actions are replaced by a learned mask token and get no loss. With separate tokens, each one predicts the next token of its own modality.
Inference. Actions are executed autoregressively; predicted observations are discarded in favor of the measured ones.

Results

Deployed zero-shot at locations around San Francisco for a week, on walkways, concrete, asphalt, tiled plazas and dirt roads.
In MuJoCo, it consistently has lower position tracking error across yaw commands than the RL policy that produced its training data (N=245).
Adding action-free trajectories lowers tracking error. Modality-aligned prediction gives 0.299 vs 0.338 tracking error without alignment; predicting states and actions gives 0.305 vs 0.335 for actions only.
Across 14 models, prediction and tracking error correlate with r=0.87. More data, longer context (16 to 48 steps) and larger models (1M to 8M parameters) each reduce tracking error. It walks backward at up to 0.5 m/s, which the action-labeled data never shows.

Limitations

The paper has no limitations section. Real-world results are qualitative only; all quantitative numbers come from simulation. Training data are flat-ground walking, and all action labels come from a single prior RL policy.

Videos

Digit walks past a street mural in San Francisco.

Digit walks across a concrete plaza by the Bay Bridge.

Figures

From Figure 2: Trajectories from neural network policies, model-based controllers, human motion capture and YouTube v... — From Figure 2: Trajectories from neural network policies, model-based controllers, human motion capture and YouTube videos train a transformer by autoregressive modeling of observations and actions, which is then deployed zero-shot around San Francisco.

From Figure 3: With complete data the transformer predicts the next observation-action pair; for MoCap and internet d... — From Figure 3: With complete data the transformer predicts the next observation-action pair; for MoCap and internet data without actions, the actions are replaced by a mask token and only the next observations are predicted.

From Figure 5: Desired (dotted) and actual (solid) base paths for the RL policy and ours, walking from the origin at ... — From Figure 5: Desired (dotted) and actual (solid) base paths for the RL policy and ours, walking from the origin at 0.5 m/s heading with yaw commands from -0.4 to 0.4 rad/s.

From Figure 7: Tracking error versus validation prediction error for 14 models; the two are linearly correlated with ... — From Figure 7: Tracking error versus validation prediction error for 14 models; the two are linearly correlated with r=0.87.

Abstract

We cast real-world humanoid control as a next token prediction problem, akin to predicting the next word in language. Our model is a causal transformer trained via autoregressive prediction of sensorimotor trajectories. To account for the multi-modal nature of the data, we perform prediction in a modality-aligned way, and for each input token predict the next token from the same modality. This general formulation enables us to leverage data with missing modalities, like video trajectories without actions. We train our model on a collection of simulated trajectories coming from prior neural network policies, model-based controllers, motion capture data, and YouTube videos of humans. We show that our model enables a full-sized humanoid to walk in San Francisco zero-shot. Our model can transfer to the real world even when trained on only 27 hours of walking data, and can generalize to commands not seen during training like walking backward. These findings suggest a promising path toward learning challenging real-world control tasks by generative modeling of sensorimotor trajectories.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong