Transporter Networks: Rearranging the Visual World for Robotic Manipulation
TL;DR Transporter Networks pick a spot, then find where it goes by matching a rotated crop of deep features around the pick against the whole top-down scene. With 100 demonstrations they reach over 90% success on 6 of 10 simulated tasks, and 98.9% on real kit assembly.
What they do
Transporter Networks treat manipulation as a sequence of spatial displacements: choose a region to pick, then decide where that region should go. The model is learned from demonstrations and uses no object models, poses or keypoints, so one architecture handles unseen objects, rope and piles of small objects, for any task built from two-pose primitives such as pick-and-place or pushes. The authors also introduce Ravens, a PyBullet benchmark of 10 tabletop tasks with scripted stochastic demonstrators.
How it works
- Spatially consistent input. RGB-D images are fused into a top-down orthographic color-plus-height map (160x320 pixels, 3.125 mm each), so objects look the same wherever they sit.
- Picking. A fully convolutional ResNet outputs a pixel-wise pick distribution, which can be multi-modal.
- Placing by transport. Two FCNs produce query and key features. The query crop around the pick is rotated into 36 angles and cross-correlated with the key features; the highest correlation gives the place pose. Training is image-wide cross-entropy on the demonstrated pick and place pixels.
- 6DoF extension. After SE(2) placement, three extra cross-correlation channels and an MLP regress roll, pitch and height.
Results
- With 100 demonstrations: 97.3% on Towers of Hanoi and 90.4% on kit assembly with unseen objects. With 1000, kits reach 94.6%, against 37.6% for Form2Fit and 11.8% for an MLP given ground-truth object poses.
- Block insertion succeeds 100% from one demonstration (Form2Fit: 17%); removing the transport step drops it to 5-8%.
- 6DoF block insertion: 91% with 1000 demonstrations, where a ground-truth-state MLP gets 5%.
- Real UR5e robots trained on teleoperation data from 13 operators: 98.9% on mouthwash kit assembly (8,141 pick-and-place actions) and 98.3% on sweeping Go pieces (6,759 pushes), averaging 20 sweeps per episode versus 18.1 for human teleoperators.
Limitations
The method is sensitive to camera-robot calibration, and it is unclear how to add torque or force actions to its spatial action space. Models are stateless and trained one per task, so non-Markovian tasks would need memory. COVID-19 lockdowns limited real-world tests to two tasks.
Videos
Figures
Abstract
Robotic manipulation can be formulated as inducing a sequence of spatial displacements: where the space being moved can encompass an object, part of an object, or end effector. In this work, we propose the Transporter Network, a simple model architecture that rearranges deep features to infer spatial displacements from visual input - which can parameterize robot actions. It makes no assumptions of objectness (e.g. canonical poses, models, or keypoints), it exploits spatial symmetries, and is orders of magnitude more sample efficient than our benchmarked alternatives in learning vision-based manipulation tasks: from stacking a pyramid of blocks, to assembling kits with unseen objects; from manipulating deformable ropes, to pushing piles of small objects with closed-loop feedback. Our method can represent complex multi-modal policy distributions and generalizes to multi-step sequential tasks, as well as 6DoF pick-and-place. Experiments on 10 simulated tasks show that it learns faster and generalizes better than a variety of end-to-end baselines, including policies that use ground-truth object poses. We validate our methods with hardware in the real world. Experiment videos and code are available at https://transporternets.github.io
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.