BridgeData V2: A Dataset for Robot Learning at Scale

Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, Sergey Levine

Seminal August 2023

Dataset clip of putting clothes into a toy laundry machine.

TL;DR A reusable real-robot dataset of 60,096 language-labeled trajectories (13 skills, 24 environments) collected with a WidowX 250 setup costing about $4,000. Six imitation and offline RL methods trained on it reach up to 0.60 average success on unseen objects and environments, and all keep non-zero success in a second lab.

What they do

BridgeData V2 is a real-robot manipulation dataset built to be reused by other labs. It has 60,096 trajectories on a low-cost WidowX 250 arm, covering 13 skills, 24 environments (mostly toy kitchens, plus tabletops, sinks and a toy laundry machine) and more than 100 objects. Every trajectory has a language label, so policies can be conditioned on goal images or instructions. The authors train six methods on it and test them on seen tasks, on unseen objects and environments, and in another lab.

How it works

Hardware. About $4,000 of off-the-shelf parts. A fixed over-the-shoulder RGBD camera, two RGB cameras re-posed every 50 trajectories and a wrist camera; 5 Hz control, teleoperated with a VR headset.
Collection. In each scene the operator performs any feasible task, with no resets between trajectories. Every 50 trajectories they also swap objects and move the workspace. Language labels are crowdsourced afterwards. 50,365 trajectories are demonstrations; 9,731 come from a heavily randomized scripted pick-and-place policy that often fails.
Methods. Goal-image BC, diffusion BC, ACT and contrastive RL, plus language-conditioned BC and RT-1. All act on the over-the-shoulder image with a 6D end-effector delta plus gripper.

Results

Seen tasks (8 tasks, 10 trials each): GCBC, diffusion BC and RT-1 each average 0.49, while language-conditioned BC averages 0.23.
Unseen objects or environments (6 tasks): GCBC 0.60, diffusion BC 0.55, CRL 0.52, RT-1 0.50, language-conditioned BC 0.08. Language policies struggle because unseen object names are not grounded in the data.
Zero-shot in a second lab, every method keeps non-zero success; RT-1 drops only from 0.47 to 0.40.
At about the same size (27-28k trajectories), training GCBC on all 13 skills instead of 3 raises unseen pick-and-place success from 0.30 to 0.65.

Limitations

The tasks are mostly low-precision and involve little force or dynamics: no throwing, heavy objects or tight insertion. All data comes from one institution and one robot model, which other labs may not want to standardize on.

Videos

Dataset clip of picking up a cup from a real dishwasher.

Dataset clip of picking an Allen key set out of a tool chest.

Figures

From Figure 1: BridgeData V2 has 60,096 trajectories across 24 environments and over 100 objects, covering skills suc... — From Figure 1: BridgeData V2 has 60,096 trajectories across 24 environments and over 100 objects, covering skills such as pick-and-place, pushing, sweeping, stacking and folding, with natural-language labels on all of the data.

From Figure 2: the WidowX 250 arm with a fixed over-the-shoulder RGBD camera and two RGB cameras whose poses are rand... — From Figure 2: the WidowX 250 arm with a fixed over-the-shoulder RGBD camera and two RGB cameras whose poses are randomized during data collection.

From Figure 5: goal-conditioned behavior cloning succeeds more often with a larger image encoder (ResNet-18, 34, 50) ... — From Figure 5: goal-conditioned behavior cloning succeeds more often with a larger image encoder (ResNet-18, 34, 50) and with a larger share of the training data, on both a seen and an unseen task.

Abstract

We introduce BridgeData V2, a large and diverse dataset of robotic manipulation behaviors designed to facilitate research on scalable robot learning. BridgeData V2 contains 60,096 trajectories collected across 24 environments on a publicly available low-cost robot. BridgeData V2 provides extensive task and environment variability, leading to skills that can generalize across environments, domains, and institutions, making the dataset a useful resource for a broad range of researchers. Additionally, the dataset is compatible with a wide variety of open-vocabulary, multi-task learning methods conditioned on goal images or natural language instructions. In our experiments, we train 6 state-of-the-art imitation learning and offline reinforcement learning methods on our dataset, and find that they succeed on a suite of tasks requiring varying amounts of generalization. We also demonstrate that the performance of these methods improves with more data and higher capacity models, and that training on a greater variety of skills leads to improved generalization. By publicly sharing BridgeData V2 and our pre-trained models, we aim to accelerate research in scalable robot learning methods. Project page at https://rail-berkeley.github.io/bridgedata

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong