Robot foundation models
Large pretrained vision-language-action policies and the data behind them.
Start here: seminal papers
RoboNet: Large-Scale Multi-Robot LearningSeminal
October 2019
RoboNet pools about 162,000 randomly collected trajectories (15 million frames) from 7 robots in 4 labs. A video-prediction model pre-trained on it and fine-tuned on 400 trajectories from a held-out Kuka or Franka reaches 40% success, beating models trained on 4.5x-20x more data from that robot alone.
Learning Transferable Visual Models From Natural Language Supervision★ Must read
February 2021
Train an image encoder and a text encoder on 400 million web (image, text) pairs to tell which caption goes with which image, then classify any dataset zero-shot by embedding its class names. CLIP reaches 76.2% zero-shot accuracy on ImageNet, matching the original ResNet-50 without using any of its labeled training images.
CLIPort: What and Where Pathways for Robotic ManipulationSeminal
September 2021
CLIPort adds a frozen CLIP stream to the Transporter pick-and-place network, so one imitation-learned policy follows language instructions on tabletop tasks. In simulation it is the only model to average above 90% on seen tasks, and a single real-robot policy learns 9 tasks from 179 image-action pairs with 55-75% success.
CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation TasksSeminal
December 2021
CALVIN is an open-source simulated benchmark in which a Franka arm must follow chains of five natural-language instructions across four related tabletop environments, learning from 24 hours of unlabeled teleoperated play. A multi-context imitation learning baseline solves 53.9% of single tasks but completes all five instructions in a row in only 0.08% of chains.
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied AgentsSeminal
January 2022
Frozen GPT-3 and Codex models write household task plans one step at a time, and a sentence-embedding model maps each step to the closest action VirtualHome can execute. No model is trained, and executability rises from 18% to 79% for Codex 12B, at some cost in human-judged correctness.
BC-Z: Zero-Shot Task Generalization with Robotic Imitation LearningSeminal
February 2022
BC-Z trains one behavior-cloning policy on 25,877 robot demonstrations of 100 manipulation tasks, conditioned on a sentence or a human video. Given only a new sentence, it succeeds at least sometimes on 24 of 29 unseen tasks, averaging 44% success on those 24.
R3M: A Universal Visual Representation for Robot ManipulationSeminal
March 2022
Pretrain an image encoder on Ego4D human videos and their language annotations, then freeze it as the perception module for behavior cloning. Across 12 simulated manipulation tasks it reaches about 62% success, more than 10% above CLIP, MoCo and ImageNet features, and on a real Franka it averages 56% versus CLIP's 24% from 20 demos per task.
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances★ Must read
April 2022
SayCan picks each next step for a mobile manipulator by multiplying a language model's score for every skill description with that skill's learned value function, so the step is both useful and feasible. With PaLM it correctly plans 84% and executes 74% of 101 kitchen instructions.
A Generalist AgentSeminal
May 2022
One 1.2B-parameter transformer, with a single set of weights, is trained to predict the next token on data from 604 control tasks plus text and images, and it plays Atari, captions images, chats and stacks blocks with a real robot arm. It scores above 50% of the expert on over 450 of the 604 tasks.
Inner Monologue: Embodied Reasoning through Planning with Language ModelsSeminal
July 2022
A frozen LLM plans robot skills step by step while success detection, scene descriptions and human answers are fed back into its prompt as text, so it can retry and replan. On a real kitchen mobile manipulator this raises success from 30.8% (SayCan) to 60.4%.
Code as Policies: Language Model Programs for Embodied ControlSeminal
September 2022
Few-shot prompt a code-writing LLM to turn language commands into Python robot policies that call perception and control APIs, with no training. On unseen simulated tabletop tasks it succeeds 80% of the time on long-horizon instructions, where CLIPort trained on 30k demonstrations gets 0%.
ProgPrompt: Generating Situated Robot Task Plans using Large Language ModelsSeminal
September 2022
Prompt an LLM with a Python-style header (the robot's actions as imports, the scene's objects as a list, a few example task programs) and have it write the new task plan as code with comments and precondition assertions. With GPT-3 it reaches 0.34 success rate on VirtualHome tasks where a prior LLM planner gets 0, and it runs on a real Franka arm for tabletop pick-and-place.
VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-TrainingSeminal
September 2022
VIP pre-trains a ResNet50 on Ego4D human videos with a goal-conditioned value objective that needs no actions, so the embedding distance to a goal image can serve as a dense reward for new robot tasks. With the frozen model as the reward, MPPI solves about 30% of FrankaKitchen tasks (44% with a larger budget), and few-shot offline RL from about 20 real demos reaches 60-100% success.
VIMA: General Robot Manipulation with Multimodal PromptsSeminal
October 2022
One transformer agent reads prompts that interleave text with images or video frames and outputs tabletop manipulation actions across 17 simulated tasks. On unseen tasks it scores up to 2.9x the success rate of the best alternative design, and with 10% of the data it still beats baselines trained on all of it.
Real-World Robot Learning with Masked Visual Pre-trainingSeminal
October 2022
Pre-train a vision transformer as a masked autoencoder on 4.5M Internet and egocentric images, freeze it, and train small behavior-cloning policies on top for real robot tasks. Across 981 real-world trials it beats CLIP by up to 75% and supervised ImageNet and from-scratch encoders by up to 81%.
RT-1: Robotics Transformer for Real-World Control at ScaleSeminal
December 2022
A 35M-parameter Transformer maps camera images and a language instruction to discretized arm and base actions at 3 Hz, trained on 130k real demonstrations of over 700 instructions. It succeeds on 97% of seen instructions and 76% of unseen ones, against at most 72% and 52% for baselines trained on the same data.
PaLM-E: An Embodied Multimodal Language ModelSeminal
March 2023
PaLM-E feeds images, state vectors and 3D object tokens into the PaLM language model, which then writes robot plans as text. It is co-trained on robot and web vision-language data. With 320 examples per task, co-training lifts TAMP planning success from about 31% to 74%, and the 562B model scores 66.1 on OK-VQA.
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object DetectionSeminal
March 2023
Grounding DINO turns the Transformer detector DINO into an open-set detector that finds any object named in a text prompt, by fusing language and image features in the neck, the query initialization and the decoder. It reaches 52.5 AP on COCO without any COCO training data and a zero-shot record of 26.1 mean AP on ODinW.
Segment Anything★ Must read
April 2023
SAM is a promptable segmentation model trained on SA-1B, 1.1B automatically generated masks on 11M images, and applied to new tasks zero-shot by prompting. From a single point it beats the strongest interactive segmenter, RITM, on 16 of 23 unseen datasets.
DINOv2: Learning Robust Visual Features without SupervisionSeminal
April 2023
A 1.1B-parameter ViT trained with self-supervision on 142M automatically curated images, then distilled into smaller models. Its frozen features reach 86.5% ImageNet-1k linear-probe accuracy, slightly ahead of OpenCLIP, and beat prior self-supervised features on segmentation, depth and retrieval.
TidyBot: Personalized Robot Assistance with Large Language ModelsSeminal
May 2023
A user gives a few example object placements, an LLM summarizes them into general rules, and a mobile manipulator uses those rules to tidy a room. The approach reaches 91.2% accuracy on unseen objects in a text benchmark, and the real robot puts away 85.0% of objects.
LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot LearningSeminal
June 2023
A simulated benchmark for lifelong robot learning, with 130 procedurally generated manipulation tasks in four suites and 50 human demos per task. In its study, plain sequential finetuning beats the lifelong-learning methods tested at forward transfer, and supervised pretraining on 90 tasks hurts later lifelong learning (AUC 0.32 to 0.22 for ER with ResNet-T).
Robots That Ask For Help: Uncertainty Alignment for Large Language Model PlannersSeminal
July 2023
KnowNo frames each step of an LLM planner as a multiple-choice question and uses conformal prediction to keep only the options it cannot rule out. When more than one option remains, the robot asks a human. It meets a user-chosen task success rate with 10-24% less human help than baselines.
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language ModelsSeminal
July 2023
GPT-4 writes code that calls open-vocabulary perception to build 3D voxel maps of where the gripper should go and what it should avoid, and a motion planner follows them in closed loop with no training. On five real-robot tasks it succeeds 88% of the time (70% under disturbances), against 24% (0%) for an LLM calling motion primitives.
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control★ Must read
July 2023
RT-2 fine-tunes vision-language models of up to 55B parameters to output robot actions as text tokens, training on web and robot data together. Over about 6,000 real trials it nearly doubles RT-1's success on unseen objects, backgrounds and environments (62% vs 32%) and scores 60% vs 17% on emergent semantic tasks.
BridgeData V2: A Dataset for Robot Learning at ScaleSeminal
August 2023
A reusable real-robot dataset of 60,096 language-labeled trajectories (13 skills, 24 environments) collected with a WidowX 250 setup costing about $4,000. Six imitation and offline RL methods trained on it reach up to 0.60 average success on unseen objects and environments, and all keep non-zero success in a second lab.
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and PlanningSeminal
September 2023
Build an open-vocabulary 3D scene graph by fusing SAM masks and CLIP features across RGB-D views, captioning each object with LLaVA and GPT-4 and letting GPT-4 label the edges, with no training. Human raters judge about 70% of node captions and 90% of edges correct, and GPT-4 planning over the graph answers negation queries far better than CLIP (R@1 0.80 vs 0.26 on Replica).
Open X-Embodiment: Robotic Learning Datasets and RT-X Models★ Must read
October 2023
Pool 60 existing robot datasets (1M+ trajectories, 22 robots) into one standard format and train RT-1 and RT-2 on a multi-robot mix. RT-1-X averages 63% success vs 41% for each dataset's own method, and RT-2-X roughly triples success on skills seen only in another robot's data.
DROID: A Large-Scale In-The-Wild Robot Manipulation DatasetSeminal
March 2024
A teleoperated manipulation dataset of 76k demonstrations across 564 scenes and 86 tasks, collected on 18 identical Franka setups at 13 institutions. Co-training a diffusion policy on DROID beats the next best training mix by 22 points of success in distribution and 17 points out of distribution.
Octo: An Open-Source Generalist Robot PolicySeminal
May 2024
Octo is an open-source transformer policy pretrained on 800k Open X-Embodiment robot episodes that follows language or goal-image commands and finetunes to new sensors, action spaces and robots. Finetuned on about 100 demos per task, it averages 72% success across six real-robot setups, versus 20% for the best baseline.
OpenVLA: An Open-Source Vision-Language-Action Model★ Must read
June 2024
An open-source 7B vision-language-action model that fine-tunes a Llama 2 VLM with fused DINOv2 and SigLIP features to output discretized robot actions, trained on 970k Open X-Embodiment trajectories. It beats the 55B RT-2-X by 16.5% absolute success across 29 tasks on two robots, and LoRA fine-tuning matches full fine-tuning while training 1.4% of the parameters.
SAM 2: Segment Anything in Images and VideosSeminal
August 2024
SAM 2 extends Segment Anything to video with a streaming memory that carries an object's mask from frame to frame, trained on a new 35.5M-mask video dataset collected with the model in the loop. It beats SAM-plus-tracker baselines with over 3x fewer interactions, and on images it is more accurate than SAM and 6x faster.
$π_0$: A Vision-Language-Action Flow Model for General Robot Control★ Must read
October 2024
π0 adds a flow-matching action expert to the PaliGemma vision-language model and pre-trains it on about 10,000 hours of data from 7 robot configurations plus open datasets, giving one 3.3B-parameter policy that controls robots at up to 50 Hz. Without fine-tuning it beats OpenVLA and Octo on every tested task, and after fine-tuning it folds laundry from a crumpled bin and assembles boxes.
FAST: Efficient Action Tokenization for Vision-Language-Action ModelsSeminal
January 2025
Compress each 1-second robot action chunk with a discrete cosine transform and byte-pair encoding before training an autoregressive VLA on it. With these FAST tokens, π0 trained on 10k hours of robot data matches the diffusion version of π0 while needing 5x fewer GPU hours to train.
GR00T N1: An Open Foundation Model for Generalist Humanoid RobotsSeminal
March 2025
An open 2.2B-parameter vision-language-action model for humanoids that pairs an Eagle-2 VLM with a flow-matching diffusion transformer, pre-trained on real robot data, human videos, simulation and generated videos. Post-trained on a real GR-1 humanoid, it averages 76.8% success versus 46.4% for Diffusion Policy, and 42.6% with only 10% of the data.
$π_{0.5}$: a Vision-Language-Action Model with Open-World Generalization★ Must read
April 2025
π0.5 extends π0 by co-training one vision-language-action model on mobile-manipulator data, other robots, subtask labels and web data, and uses it both to pick the next subtask and to output actions. Although 97.6% of its pre-training examples come from sources other than household mobile manipulation, it cleans kitchens and bedrooms in homes it never saw.
Recent
OopsieRecover: Learning Transferable VLA Corrections from Semantic Interventions
November 2026
OopsieRecover turns a handful of human interventions, each with a short note on why, into gated costs over 3D point sets that decide when and how to steer a frozen pi0.5 VLA. On a real DROID setup it raises task progress from 35.1% for the next-best method to 61.0%, and the corrections carry over to new object poses and scenes.
Failure Prediction and Classification in Vision-Language-Action Models Via Calibrated Latent Probing
November 2026
VLA-Tug is a small LSTM probe on a VLA's backbone embeddings that predicts failures before they happen and names the likely failure mode, with an alarm threshold calibrated to bound both false alarms and missed early detections. On held-out LIBERO-X skills it raises early-detection area by 30% over the strongest learned baseline, at 0.12 ms per step.
Spatial-UMI-Dream: Scaling World Action Models with Metric Geometry for UMI Manipulation
November 2026
Spatial-UMI-Dream pretrains a 5B world action model on 605 hours of UMI handheld-gripper data to predict future RGB, stereo-derived metric depth and actions together. It reaches 85.0% average success on six real bimanual tasks, 24.2 points above the best baseline (π0.5).
Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement
November 2026
They post-train the pi0.5 VLA online with RL on up to 144 simulated pick tasks at once and test on held-out scenes and objects. With critic-based recipes the gains grow with the number of training tasks, up to +58 points on unseen scenes and +39 on unseen objects, while filtered fine-tuning generalizes much less.
GSBR: Geometric Support Behavior Regularization for Online RL
November 2026
GSBR fine-tunes a frozen imitation-learned base policy with online RL, penalizing actions only by how far they fall outside the base policy's estimated support. Across 19 simulated tasks on four benchmarks it raises average success from 33% for the base policy to 95%, with about 23% higher normalized throughput than the next-best RL method.
Asynchronous Robot Policies
November 2026
ARP splits a vision-language-action policy into two processes. A slow VLM keeps refreshing a compact memory, while a small vision-action expert reads it and acts at a much higher rate, trained to cope with stale memory. On LIBERO it keeps 81.8% success with the VLM updating at only 0.5 Hz, and when the target shifts 5 cm mid-reach it succeeds 51.5% of the time against 9.5% for a π0-style baseline.
AIRoA-0: Long-Horizon, Precise, Cross-Embodiment Manipulation from Handheld Data
November 2026
Pretrain a π0.5-based policy on 16,151 hours of handheld-gripper demonstrations with no robot data at all, then post-train it per task on a few hours of handheld data. On ten long bimanual tasks at a site outside its data, A-0 averages 78% success against 27% for the best of four post-trained VLA baselines.
Robot-Matched Soft Fingers for UMI Data Collection: Lessons from 100 Hours of Pick and Place
November 2026
A battery-powered handheld UMI with soft, angled fingers that match the robot's gripper, used to collect 99.7 hours of pick and place over 53 objects. Mid-training π0.5 on this data raises success after fine-tuning on unseen objects from 35% to 57% on a Galaxea R1 Pro.
ForceLDM: A Force-aware Latent Dynamics Model for Contact-Rich Manipulation
November 2026
ForceLDM distills future force/torque and optical-flow features from a privileged teacher into a π0-based student policy that sees only current observations. On five real-robot contact-rich tasks it averages 87.1% success, 14.9 points above the strongest baseline.
HeLM: Hierarchical Explicit Language Memory for Non-Markovian Manipulation
November 2026
A fine-tuned VLM keeps a running text memory of task events, such as which buttons were already pressed, and feeds commands to a π0 low-level policy, so the robot can count, follow orders and recall information across instructions. On 10 real-robot memory tasks it reaches 85.7% subtask success within episodes (vs 72.9% for the best baseline) and 89.2% across episodes.
New papers stay for 14 days.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.