TossingBot: Learning to Throw Arbitrary Objects with Residual Physics

Andy Zeng, Shuran Song, Johnny Lee, Alberto Rodriguez, Thomas Funkhouser

Seminal March 2019

TL;DR TossingBot learns by trial and error to grasp arbitrary objects from a bin and throw them into boxes beyond a UR5's reach, with a network that predicts a correction on top of a ballistic release velocity. On a real robot it reaches 514 picks per hour with 84.7% throwing accuracy on seen objects (82.3% on unseen).

What they do

Throwing lets a picking arm place objects into boxes outside its kinematic reach without carrying them there. But where an object lands depends on how it was grasped, its mass distribution and aerodynamics. TossingBot learns grasping and throwing jointly from RGB-D images by trial and error. A ballistic model supplies the throw, and a network learns a residual on it.

How it works

Grasping. An RGB-D heightmap of the bin (5 mm pixels) is rotated to 16 orientations; a fully convolutional network predicts the success probability of a top-down parallel-jaw grasp at every pixel and angle.
Throwing. Release height, release distance and a 45° launch angle are fixed, so only the release speed is unknown. Projectile equations for a point mass give a speed estimate; a second network predicts a per-grasp residual that is added to it.
Joint self-supervision. A grasp counts as successful only if the throw lands in the right box. An overhead camera finds the actual landing box, which gives the residual label. Boxes lift to slide objects back into the bin, so training needs little human reset.

Results

Real UR5, 80+ objects, 12 boxes: throwing success 84.7% (seen) and 82.3% (unseen) versus 61.3/58.5% for physics-only and 54.2/52.0% for a regression baseline pretrained on physics. 15 untrained people scored 80.1 ± 10.8% on the unseen objects.
608 grasps per hour at 84.7% accuracy gives 514 picks per hour, versus 432 for the same system placing objects and 312 for Dex-Net 4.0 (not a like-for-like comparison).
New box locations (real): 83.9% versus 62.2% physics-only and 32.7% regression.
Supervising grasps with throw accuracy raises throwing success on hammers in simulation over grasp-width supervision.

Limitations

Objects must be rigid and survive being thrown. Object properties are inferred from vision alone, with no force or tactile sensing. Only the landing location is controlled, not the object’s pose in flight, and residual physics is shown only for throwing.

Videos

Figures

From Figure 1: TossingBot grasps objects from an unstructured bin and throws them into target boxes outside its maxim... — From Figure 1: TossingBot grasps objects from an unstructured bin and throws them into target boxes outside its maximum reach range.

From Figure 3: Ways to choose an action: (a) analytical model, (b) learned policy, (c) hybrid model that learns resid... — From Figure 3: Ways to choose an action: (a) analytical model, (b) learned policy, (c) hybrid model that learns residuals on predicted future states, (d) hybrid policy, used here, that adds a learned residual to the analytical action.

From Figure 4: An RGB-D heightmap is turned into spatial features while a ballistic controller estimates the release ... — From Figure 4: An RGB-D heightmap is turned into spatial features while a ballistic controller estimates the release velocity for the target. The grasping module scores grasps per pixel over 16 rotations; the throwing module predicts a residual velocity for each grasp, added to the ballistic estimate.

From Figure 7: Throwing success over training on the Hammers set in simulation; Residual-physics ends above Physics-o... — From Figure 7: Throwing success over training on the Hammers set in simulation; Residual-physics ends above Physics-only, Regression-PoP and Regression.

Abstract

We investigate whether a robot arm can learn to pick and throw arbitrary objects into selected boxes quickly and accurately. Throwing has the potential to increase the physical reachability and picking speed of a robot arm. However, precisely throwing arbitrary objects in unstructured settings presents many challenges: from acquiring reliable pre-throw conditions (e.g. initial pose of object in manipulator) to handling varying object-centric properties (e.g. mass distribution, friction, shape) and dynamics (e.g. aerodynamics). In this work, we propose an end-to-end formulation that jointly learns to infer control parameters for grasping and throwing motion primitives from visual observations (images of arbitrary objects in a bin) through trial and error. Within this formulation, we investigate the synergies between grasping and throwing (i.e., learning grasps that enable more accurate throws) and between simulation and deep learning (i.e., using deep networks to predict residuals on top of control parameters predicted by a physics simulator). The resulting system, TossingBot, is able to grasp and throw arbitrary objects into boxes located outside its maximum reach range at 500+ mean picks per hour (600+ grasps per hour with 85% throwing accuracy); and generalizes to new objects and target locations. Videos are available at https://tossingbot.cs.princeton.edu

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong