Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich

★ Must read July 2023

The mobile manipulator moves a banana onto the paper marked "3" (2x speed).

TL;DR RT-2 fine-tunes vision-language models of up to 55B parameters to output robot actions as text tokens, training on web and robot data together. Over about 6,000 real trials it nearly doubles RT-1's success on unseen objects, backgrounds and environments (62% vs 32%) and scores 60% vs 17% on emergent semantic tasks.

What they do

RT-2 turns a large pretrained vision-language model (VLM) directly into a closed-loop robot policy. Robot actions are written as strings of integers and treated as text, so the VLM is fine-tuned on robot demonstrations without adding any new layers. The question is whether web pretraining helps it generalize and follow instructions absent from the robot data.

How it works

  • Actions as tokens. End-effector position and rotation deltas and gripper extension are each discretized into 256 bins, plus a terminate flag: 8 integers per step. PaLI-X reuses its integer tokens; PaLM-E overwrites its 256 least-used tokens.
  • Co-fine-tuning. PaLI-X (5B, 55B) and PaLM-E (12B) are trained on their original web data (VQA, captioning) mixed with the RT-1 dataset (13 robots, 17 months, office kitchen).
  • Cloud inference. The model runs on a multi-TPU cloud service queried over the network: 1-3 Hz for 55B, about 5 Hz for 5B.
  • Chain of thought. A PaLM-E variant is fine-tuned for a few hundred steps to write a natural-language “Plan” before the action tokens.

Results

  • About 6,000 real trials. Seen tasks: RT-2 91-93% vs 92% for RT-1. Unseen objects, backgrounds and environments: both RT-2 models average 62%, vs 32% for RT-1 and 35% for MOO.
  • Emergent tasks (symbols, math, logos, multilingual commands, people): RT-2-PaLI-X-55B averages 60% vs 17% for RT-1.
  • Ablations on unseen scenes: 5B trained from scratch gets 9%, robot-only fine-tuning 42%, co-fine-tuning 44%; at 55B, 52% vs 63%.
  • Simulated Language-Table: a 3B PaLI-based RT-2 reaches 90% vs 77% for LAVA and 74% for RT-1.

Limitations

Web pretraining adds no new motions: the robot only redeploys skills present in its demonstrations, and it does poorly at grasping objects by specific parts, wiping, tool use, folding and reasoning through several layers of indirection. The models are costly to run, which may limit high-frequency control, and few VLMs are available for this kind of fine-tuning.

Videos

Following "move the green objects together" at an office desk (2x speed).
Language-Table arm pushing a block to named condiment bottles (4x speed).

Figures

From Figure 1: RT-2 overview. Robot actions are cast into text tokens and trained together with Internet-scale vision...
From Figure 1: RT-2 overview. Robot actions are cast into text tokens and trained together with Internet-scale vision-language data; at inference the tokens are de-tokenized into robot actions for closed-loop control.
From Figure 2: RT-2 generalizing to real-world instructions that require reasoning, symbol understanding and human re...
From Figure 2: RT-2 generalizing to real-world instructions that require reasoning, symbol understanding and human recognition.
From Figure 4: Success rates of two RT-2 instantiations and baselines on seen tasks and on unseen objects, background...
From Figure 4: Success rates of two RT-2 instantiations and baselines on seen tasks and on unseen objects, backgrounds and environments.
From Figure 8: Success rates of RT-2 and two baselines on the emergent skill evaluations: symbol understanding, reaso...
From Figure 8: Success rates of RT-2 and two baselines on the emergent skill evaluations: symbol understanding, reasoning and human recognition.
Abstract

We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control to boost generalization and enable emergent semantic reasoning. Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web. To this end, we propose to co-fine-tune state-of-the-art vision-language models on both robotic trajectory data and Internet-scale vision-language tasks, such as visual question answering. In contrast to other approaches, we propose a simple, general recipe to achieve this goal: in order to fit both natural language responses and robotic actions into the same format, we express the actions as text tokens and incorporate them directly into the training set of the model in the same way as natural language tokens. We refer to such category of models as vision-language-action models (VLA) and instantiate an example of such a model, which we call RT-2. Our extensive evaluation (6k evaluation trials) shows that our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training. This includes significantly improved generalization to novel objects, the ability to interpret commands not present in the robot training data (such as placing an object onto a particular number or icon), and the ability to perform rudimentary reasoning in response to user commands (such as picking up the smallest or largest object, or the one closest to another object). We further show that incorporating chain of thought reasoning allows RT-2 to perform multi-stage semantic reasoning, for example figuring out which object to pick up for use as an improvised hammer (a rock), or which type of drink is best suited for someone who is tired (an energy drink).

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.