Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

GR00T N1: An Open Foundation Model for Generalist Humanoid Robots

NVIDIA, Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xinye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang, Zu Wang, Jing Wang, Qi Wang, Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhenjia Xu, Seonghyeon Ye, Zhiding Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, Yuke Zhu

Seminal March 2025

TL;DR An open 2.2B-parameter vision-language-action model for humanoids that pairs an Eagle-2 VLM with a flow-matching diffusion transformer, pre-trained on real robot data, human videos, simulation and generated videos. Post-trained on a real GR-1 humanoid, it averages 76.8% success versus 46.4% for Diffusion Policy, and 42.6% with only 10% of the data.

What they do

GR00T N1 is an open vision-language-action (VLA) model intended as one foundation policy for humanoids and other robots. Real humanoid data is scarce, so they train on a “data pyramid”: web data and human egocentric videos at the base, simulated and video-model-generated trajectories in the middle, and real teleoperation data at the top. The pretrained model is then post-trained per embodiment on a small task dataset.

How it works

  • Two systems. System 2 is the Eagle-2 VLM, which encodes the camera image and instruction; features come from its 12th LLM layer. System 1 is a diffusion transformer trained with flow matching that cross-attends to those features and outputs 16-step action chunks in 4 denoising steps (63.9 ms on an L40).
  • Cross-embodiment. Per-embodiment MLPs encode state and decode actions, so one set of weights covers single arms, bimanual arms and the GR-1 humanoid.
  • Data without actions. Human and generated videos get action labels from a VQ-VAE latent-action model, treated as its own “embodiment”, or from an inverse dynamics model trained on robot data.
  • Synthetic data. A fine-tuned image-to-video model turns 88 h of teleoperation into 827 h of new trajectories; DexMimicGen generates 780k simulation trajectories in 11 h.

Results

  • Simulation, 100 demos per task: 45.0% average success vs 33.4% for Diffusion Policy and 26.4% for BC-Transformer. The largest gap is on GR-1 tabletop tasks (50.0% vs 32.7%).
  • Real GR-1: 76.8% average vs 46.4% for Diffusion Policy with full data; with 10% of the data, 42.6% vs 10.2%.
  • With no post-training, the pretrained model scores 11.5/15 on a task that needs a left-to-right handover and 11/15 placing novel objects in an unseen container.
  • Co-training on generated videos raises RoboCasa success by 4.2-8.8% across data regimes and real GR-1 success by 5.8% on average over 8 tasks.

Limitations

Only short-horizon tabletop manipulation; long-horizon loco-manipulation is left to future work. Generated and simulated data still struggle to be diverse and counterfactual while obeying physics. Post-training can erase pretrained skills: after post-training on right-hand-only demos, the model loses the handover behavior the pretrained checkpoint showed.

Figures

From Figure 1: the data pyramid. Web data and human videos form the base, synthetic data the middle and real-robot da...
From Figure 1: the data pyramid. Web data and human videos form the base, synthetic data the middle and real-robot data the top; data quantity falls and embodiment specificity rises toward the top.
From Figure 3: the Eagle-2 VLM encodes the image and instruction; DiT blocks with embodiment-specific state and actio...
From Figure 3: the Eagle-2 VLM encodes the image and instruction; DiT blocks with embodiment-specific state and action encoders cross-attend to those tokens and denoise a chunk of actions.
From Figure 10: average simulation success of Diffusion Policy and GR00T-N1-2B on RoboCasa, DexMG and GR-1 with 30, 1...
From Figure 10: average simulation success of Diffusion Policy and GR00T-N1-2B on RoboCasa, DexMG and GR-1 with 30, 100 and 300 demos per task.
From Figure 12: on the real GR-1, post-trained GR00T-N1-2B completes cucumber and lemon pick-and-place tasks where Di...
From Figure 12: on the real GR-1, post-trained GR00T-N1-2B completes cucumber and lemon pick-and-place tasks where Diffusion Policy grasps inaccurately or stays stuck.
Abstract

General-purpose robots need a versatile body and an intelligent mind. Recent advancements in humanoid robots have shown great promise as a hardware platform for building generalist autonomy in the human world. A robot foundation model, trained on massive and diverse data sources, is essential for enabling the robots to reason about novel situations, robustly handle real-world variability, and rapidly learn new tasks. To this end, we introduce GR00T N1, an open foundation model for humanoid robots. GR00T N1 is a Vision-Language-Action (VLA) model with a dual-system architecture. The vision-language module (System 2) interprets the environment through vision and language instructions. The subsequent diffusion transformer module (System 1) generates fluid motor actions in real time. Both modules are tightly coupled and jointly trained end-to-end. We train GR00T N1 with a heterogeneous mixture of real-robot trajectories, human videos, and synthetically generated datasets. We show that our generalist robot model GR00T N1 outperforms the state-of-the-art imitation learning baselines on standard simulation benchmarks across multiple robot embodiments. Furthermore, we deploy our model on the Fourier GR-1 humanoid robot for language-conditioned bimanual manipulation tasks, achieving strong performance with high data efficiency.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.