RoboNet: Large-Scale Multi-Robot Learning

Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, Chelsea Finn

Seminal October 2019

A RoboNet trajectory from the Franka arm in a bin of plush toys.

TL;DR RoboNet pools about 162,000 randomly collected trajectories (15 million frames) from 7 robots in 4 labs. A video-prediction model pre-trained on it and fine-tuned on 400 trajectories from a held-out Kuka or Franka reaches 40% success, beating models trained on 4.5x-20x more data from that robot alone.

What they do

Most robot-learning experiments use one robot in one lab. RoboNet is an open dataset of about 162,000 trajectories (nearly 15 million frames) from 7 robots, 4 institutions and 113 camera configurations. The data was collected autonomously by executing random actions. The authors then test whether models trained on this pooled data generalize to new viewpoints, scenes, grippers and robots.

How it works

Shared action space. Every robot is commanded with end-effector deltas (XYZ, yaw, binary gripper) in its own base frame. Cameras are not calibrated, so the model has to infer the camera-robot relation from two context frames.
Data collection. Actions are sampled from per-robot Gaussians, plus a grasp primitive that closes the gripper near the table. The authors collected 104.4k trajectories and added 5k from a Fetch and 56k from a Google robot.
Visual foresight. An action-conditioned video-prediction model predicts flow fields that move user-designated pixels. MPPI planning (600 samples, horizon 15) picks actions that push those pixels to the goal, replanning every step.
Inverse model. A one-step model that predicts the action leading from the current image to a goal image.

Results

Viewpoints: a model trained on 90 viewpoints ends 16.2 cm from the goal from a held-out view, against 23.2 cm for a single-view model.
New robots, 400 fine-tuning trajectories: 40% success on the Kuka (scratch: 10%; 30% with 1,800 trajectories) and 40% on the Franka (scratch: 20%; 35% with 8,000).
On the Baxter, pre-training on Sawyer data alone (83%) beats pre-training on all of RoboNet (58%).
Capacity: per-pixel L1 error falls from 0.104 at 200M parameters to 0.085 at 500M, and even the 500M model still underfits.

Limitations

The tasks are simple, low-precision pushing and pick-and-place. Random-policy data rules out more complex interactions. Because the models underfit, more diverse pre-training data can hurt performance. Robots must stop after every action. The inverse model handles reaching but does poorly once objects are involved.

Videos

A RoboNet trajectory from the Kuka arm in a bin of plastic objects.

A RoboNet trajectory from the low-cost WidowX arm.

Figures

From Figure 1: Example RoboNet trajectories over time from the Sawyer, Franka, WidowX, Kuka, Baxter, Google and Fetch... — From Figure 1: Example RoboNet trajectories over time from the Sawyer, Franka, WidowX, Kuka, Baxter, Google and Fetch robots.

From Figure 5: The recurrent video-prediction model. At each step it takes an action and the current image, updates i... — From Figure 5: The recurrent video-prediction model. At each step it takes an action and the current image, updates its LSTM state and predicts a flow field that warps the image into the next predicted frame.

From Figure 4: Visual foresight on the Franka, pre-trained on RoboNet without Franka data and fine-tuned on 400 Frank... — From Figure 4: Visual foresight on the Franka, pre-trained on RoboNet without Franka data and fine-tuned on 400 Franka trajectories, grasps a thin plastic cup and moves it to the goal. Predicted and actual trajectories are shown.

Abstract

Robot learning has emerged as a promising tool for taming the complexity and diversity of the real world. Methods based on high-capacity models, such as deep networks, hold the promise of providing effective generalization to a wide range of open-world environments. However, these same methods typically require large amounts of diverse training data to generalize effectively. In contrast, most robotic learning experiments are small-scale, single-domain, and single-robot. This leads to a frequent tension in robotic learning: how can we learn generalizable robotic controllers without having to collect impractically large amounts of data for each separate experiment? In this paper, we propose RoboNet, an open database for sharing robotic experience, which provides an initial pool of 15 million video frames, from 7 different robot platforms, and study how it can be used to learn generalizable models for vision-based robotic manipulation. We combine the dataset with two different learning algorithms: visual foresight, which uses forward video prediction models, and supervised inverse models. Our experiments test the learned algorithms' ability to work across new objects, new tasks, new scenes, new camera viewpoints, new grippers, or even entirely new robots. In our final experiment, we find that by pre-training on RoboNet and fine-tuning on data from a held-out Franka or Kuka robot, we can exceed the performance of a robot-specific training approach that uses 4x-20x more data. For videos and data, see the project webpage: https://www.robonet.wiki/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong