Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation

Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B. Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, Vincent Sitzmann

Seminal December 2021

The mug-hanging task carried over to a tall mug unlike the demonstration mugs.

TL;DR Describe an object by an SE(3)-equivariant neural field of 3D descriptors, learned by self-supervised shape reconstruction, and transfer grasp and place poses from 10 demonstrations by matching those descriptors. In simulation, mug pick-and-place on unseen mugs succeeds 88% of the time upright and 58% in arbitrary poses, against 45% and 17% for a 2D-descriptor baseline.

What they do

Learning pick-and-place from demonstrations usually needs demos that cover many object instances and poses. Neural Descriptor Fields (NDFs) let a Franka Panda repeat a task (hang a mug on a rack, place a bowl or bottle upright on a shelf) on new objects of the same category in arbitrary 6-DoF poses, from 10 demonstrations on upright objects and without hand-labeled keypoints.

How it works

Point descriptors. An occupancy network with a PointNet encoder is pretrained to reconstruct full shapes from partial point clouds. The descriptor of any 3D point, on or off the object, is the concatenation of the network’s activations across all layers. Mean-centering plus Vector Neurons make it SE(3)-equivariant.
Pose descriptors. The pose of a gripper, rack or shelf is represented by a rigid set of query points sampled in that body’s bounding box; their point descriptors are concatenated.
Transfer. Grasp and place descriptors are averaged over the demos. For a new point cloud, Adam searches from a random start for the pose minimizing the L1 descriptor distance; IK and motion planning execute it.

Results

Simulation (PyBullet, ShapeNet objects, 10 demos), overall success on unseen upright objects: mugs 0.88, bowls 0.91, bottles 0.87, versus 0.45, 0.11 and 0.24 for a Dense Object Nets (DON) pipeline.
Upright demos, test objects in arbitrary poses: NDF 0.58, 0.78 and 0.77; DON 0.17, 0.00 and 0.01.
Mug ablations: random-weight features score 0.00 overall, first- or last-layer features alone 0.65, all layers 0.88. Shrinking or enlarging the query point set relative to the interacting body’s bounding box (0.1x to 10x) drops overall success to 0.27-0.63.
A 6-DoF extension of Transporter Nets stayed below 10%. Real-robot runs on new mugs, bowls and bottles in challenging poses are shown qualitatively, without success rates.

Limitations

Needs a segmented object point cloud and a placement target that stays fixed between demos and test. The authors attribute the drop in arbitrary poses to the PointNet encoder not being robust to unseen occlusions, such as a mug’s bottom when demos were upright. Only rigid objects were tested, and NDFs transfer single poses, not trajectories.

Videos

Picking and hanging a mug lying on its side, a pose absent from the upright demos.

Bowl task, one upright demonstration beside new bowls in tilted poses, 8x speed.

Figures

From Figure 4: Given a demonstration point cloud and a target point, NDFs define an energy over 3D points on an unsee... — From Figure 4: Given a demonstration point cloud and a target point, NDFs define an energy over 3D points on an unseen object whose minimum is the matching point. The energy is differentiable in the point coordinates.

From Figure 7: From a demonstration point cloud and gripper pose (left), the grasp pose (orange) on an unseen mug (bl... — From Figure 7: From a demonstration point cloud and gripper pose (left), the grasp pose (orange) on an unseen mug (blue) is found by minimizing the difference between the descriptors of the gripper query points (green) in the demo and at test time.

From Figure 8: Both DON and NDF find a grasp on an upright mug. On a mug in an arbitrary pose, DON detects the wrong ... — From Figure 8: Both DON and NDF find a grasp on an upright mug. On a mug in an arbitrary pose, DON detects the wrong keypoint while NDF still infers a grasp.

From Figure 9: Real-robot pick-and-place of unseen bottles and bowls in out-of-distribution poses, after ten upright ... — From Figure 9: Real-robot pick-and-place of unseen bottles and bowls in out-of-distribution poses, after ten upright demonstrations per category.

Abstract

We present Neural Descriptor Fields (NDFs), an object representation that encodes both points and relative poses between an object and a target (such as a robot gripper or a rack used for hanging) via category-level descriptors. We employ this representation for object manipulation, where given a task demonstration, we want to repeat the same task on a new object instance from the same category. We propose to achieve this objective by searching (via optimization) for the pose whose descriptor matches that observed in the demonstration. NDFs are conveniently trained in a self-supervised fashion via a 3D auto-encoding task that does not rely on expert-labeled keypoints. Further, NDFs are SE(3)-equivariant, guaranteeing performance that generalizes across all possible 3D object translations and rotations. We demonstrate learning of manipulation tasks from few (5-10) demonstrations both in simulation and on a real robot. Our performance generalizes across both object instances and 6-DoF object poses, and significantly outperforms a recent baseline that relies on 2D descriptors. Project website: https://yilundu.github.io/ndf/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong