RT-1: Robotics Transformer for Real-World Control at Scale

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich

Seminal December 2022

Unseen kitchen, 4x speed, opening a drawer to fetch rice chips.

TL;DR A 35M-parameter Transformer maps camera images and a language instruction to discretized arm and base actions at 3 Hz, trained on 130k real demonstrations of over 700 instructions. It succeeds on 97% of seen instructions and 76% of unseen ones, against at most 72% and 52% for baselines trained on the same data.

What they do

RT-1 is a single language-conditioned policy for a mobile manipulator (7-DoF arm, gripper, mobile base), trained by behavior cloning on about 130k human demonstrations. The data was collected by 13 robots over 17 months in office kitchens and covers over 700 instructions, such as picking, placing and opening drawers.

How it works

Tokenizer. A history of 6 images (300x300) goes through an ImageNet-pretrained EfficientNet-B3. FiLM layers, initialized to identity, condition it on a Universal Sentence Encoder embedding of the instruction, giving 81 tokens per image.
Compression. TokenLearner reduces these to 8 tokens per image, 48 in total, which feed an 8-layer decoder-only Transformer (35M parameters for the whole model).
Actions. 11 dimensions (arm pose and gripper, base motion, and a mode switch), each discretized into 256 bins and trained with cross-entropy. Actions are not generated autoregressively (that slowed inference over 2x with no gain), which keeps closed-loop control at 3 Hz.

Results

Over 3000 real-world trials against Gato and BC-Z trained on the same data: 97% on seen instructions (best baseline 72%), 76% on unseen instructions (52%), 83% with distractors (47%) and 59% on new backgrounds (41%).
Inside SayCan, long-horizon execution succeeds 67% of the time in both kitchens. BC-Z gets 53% and 13%, and Gato 33% and 0%.
Adding simulation data raises success on objects seen only in simulation from 23% to 87%. Adding Kuka bin-picking data raises bin-picking success from 22% to 39%. Both cost 2 points on the original tasks.
Data diversity matters more than quantity: dropping 25% of tasks while keeping 97% of the data hurts generalization about as much as cutting the data by 49%.

Limitations

RT-1 is imitation learning, so it may not surpass its demonstrators. It generalizes only to new combinations of seen concepts, not to new motions. The tasks are not very dexterous, and robustness to new backgrounds (59%) still lags.

Videos

Multi-step request at 4x speed, taking the chips out of a drawer.

Long task at 20x speed, bringing all graspable objects from a counter.

Figures

From Figure 1: RT-1 is trained on 700+ tasks and 130k demonstrations, and evaluated on new tasks, new backgrounds and... — From Figure 1: RT-1 is trained on 700+ tasks and 130k demonstrations, and evaluated on new tasks, new backgrounds and distractors, long-horizon tasks and new data sources.

From Figure 3: The instruction embedding conditions an ImageNet-pretrained EfficientNet through FiLM layers. TokenLea... — From Figure 3: The instruction embedding conditions an ImageNet-pretrained EfficientNet through FiLM layers. TokenLearner cuts each image's 81 vision-language tokens to 8, and a decoder-only Transformer over 6 images outputs discretized actions.

From Tables 2, 4 and 5: Success of RT-1 and baselines on seen tasks, unseen tasks, distractors and backgrounds (left)... — From Tables 2, 4 and 5: Success of RT-1 and baselines on seen tasks, unseen tasks, distractors and backgrounds (left), and the change in RT-1's success from adding simulation data (middle) or Kuka bin-picking data (right).

Abstract

By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance. While this capability has been demonstrated in other fields such as computer vision, natural language processing or speech recognition, it remains to be shown in robotics, where the generalization capabilities of the models are particularly critical due to the difficulty of collecting real-world robotic data. We argue that one of the keys to the success of such general robotic models lies with open-ended task-agnostic training, combined with high-capacity architectures that can absorb all of the diverse, robotic data. In this paper, we present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks. The project's website and videos can be found at robotics-transformer1.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong