A WidowX arm manipulates a pepper in a toy kitchen sink.
TL;DR Pool 60 existing robot datasets (1M+ trajectories, 22 robots) into one standard format and train RT-1 and RT-2 on a multi-robot mix. RT-1-X averages 63% success vs 41% for each dataset's own method, and RT-2-X roughly triples success on skills seen only in another robot's data.
What they do
Robot learning usually trains a separate policy per robot and setup. This paper tests whether one policy trained on many robots’ data can beat those specialists. Twenty-one institutions pooled 60 existing datasets from 34 labs into the Open X-Embodiment dataset: over 1M real trajectories from 22 embodiments, all converted to the RLDS format. Existing RT-1 and RT-2 models trained on a 9-manipulator subset become RT-X.
How it works
Coarse alignment. One canonical camera view per dataset, resized to a common resolution. Actions become a 7-DoF end-effector vector, normalized per dataset and split into 256 bins. Coordinate frames are not aligned, so the same action can move different robots differently.
RT-1-X. The 35M-parameter RT-1: a FiLM-conditioned EfficientNet over 15 images plus the instruction, feeding a decoder-only Transformer.
RT-2-X. RT-2-PaLI-X, a vision-language model that outputs actions as text tokens, co-fine-tuned on a roughly 1:1 mix of web vision-language data and robot data.
No embodiment-specific mechanisms. Evaluation: 3600 trials on 6 robots.
Results
Five small-data domains: RT-1-X beats the original method on 4 of 5, with mean success of 63% vs 41% (original) and 44% (RT-1 on that dataset alone).
Large-data domains: RT-1-X underfits (73% vs 92% for RT-1 on the Google Robot). The 55B RT-2-X scores 91% there and 50% vs 40% on Bridge at Stanford.
Google Robot skills that appear only in the WidowX Bridge data: RT-2-X 75.8% vs RT-2 27.3%. Removing Bridge drops it to 42.8%. On unseen objects and backgrounds the two are on par (61% vs 62%).
Ablations: a 5B model reaches 44.4% on those skills, 14.5% without image history and 0% without web pretraining.
Limitations
Experiments use 9 of the 22 embodiments, all with similar sensing and actuation. The paper does not test transfer to new robots and gives no rule for when positive transfer happens.
Videos
A robot routes a cable into a clip on a tabletop.A robot arm pulls open a kitchen cabinet door.
Figures
From Figure 1: An open, large-scale robot learning dataset curated from 21 institutions, spanning diverse behaviors, robot embodiments and environments.From Figure 2: RT-1-X and RT-2-X both take images and a text instruction and output discretized end-effector actions. RT-1-X uses a FiLM-conditioned EfficientNet and a Transformer; RT-2-X builds on a vision-language model and represents actions as text tokens.From Figure 3: Success rates on five small-data domains. RT-1-X mean success is 50% higher than the Original Method or RT-1; RT-1 and RT-1-X share an architecture, so the gain comes from co-training on the multi-robot data.Abstract
Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train generalist X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. More details can be found on the project website https://robotics-transformer-x.github.io.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
