Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation

Peter R. Florence, Lucas Manuelli, Russ Tedrake

Seminal June 2018

TL;DR A network trained by self-supervision on robot-collected RGBD scans gives every pixel a descriptor. The descriptor for a given point stays the same across views and deformations, and optionally across instances of a class. It trains in about 20 minutes per object (47 objects tested) and lets a robot grasp a point that a user clicked in a single reference image.

What they do

Dense Object Nets map an RGB image to a D-dimensional descriptor at every pixel. The same physical point on an object should get the same descriptor across viewpoints and deformations, and optionally across instances of a class. Training needs no human labels: a robot-mounted RGBD camera scans static scenes, and 3D reconstructions supply the pixel correspondences. A robot then uses the descriptors to grasp a point that a user clicked in one reference image.

How it works

Self-supervised correspondences. A KUKA iiwa with a wrist-mounted Primesense camera scans each tabletop scene in about 70 s. TSDF fusion, using camera poses from forward kinematics, builds a dense reconstruction. Two pixels are a match if they land on the same reconstruction vertex. A 34-layer ResNet is trained on these matches with a pixelwise contrastive loss.
Object-centric training. 3D change detection produces object masks without labels. Matches are sampled only on the object, backgrounds are domain-randomized, and the non-match loss is scaled by the number of hard negatives.
Multi-object descriptors. A cross-object loss treats every pixel pair between two different objects as a non-match. Training also uses cluttered scenes and synthetic scenes built by layering object masks.
Grasping. The pixel whose descriptor best matches the clicked point is looked up in the point cloud. A geometric antipodal grasp is then sampled around that point.

Results

Tested on 47 objects in 275 scenes, including hats, shoes, mugs and low-texture toys. Training a new object takes about 20 minutes: 3500 steps of about 0.22 s each, plus collecting a few scenes.
On human-labeled correspondences, the full procedure’s best match lies within 13% of the image diagonal in 93% of image pairs. Removing object masking or hard-negative scaling hurts accuracy, and Schmidt et al.’s procedure does significantly worse.
Descriptors trained on 6 hats generalize to unseen hats, including a blue one, a color never seen in training. The cross-object loss separates objects in descriptor space. It costs some precision when descriptors have few dimensions, and adding dimensions recovers it.
Demonstrations: grasping the caterpillar’s tail or right ear across deformed configurations, grasping the same point on unseen shoes, and grasping a specific shoe’s heel in a cluttered pile.

Limitations

The grasps are demonstrations; the paper reports no success rates. Class generalization fails for objects that differ enough: two anthropomorphic toys get no shared descriptors. For many scenes a human moved the object between scans, and a human must indicate which scenes contain which object. The descriptors give a grasp point but not the gripper orientation, which comes from a geometric planner.

Videos

Figures

From Figure 1: Data collection and training. (a) a robot arm collects the data; (b) change detection on the dense 3D ... — From Figure 1: Data collection and training. (a) a robot arm collects the data; (b) change detection on the dense 3D reconstruction isolates the object; (c)-(f) matches in green and non-matches in red for background randomization, cross-object loss, direct multi-object and synthetic multi-object training.

From Figure 2: RGB frames and the descriptor images the network outputs, consistent across deformation of a plush cat... — From Figure 2: RGB frames and the descriptor images the network outputs, consistent across deformation of a plush caterpillar (a) and across instances of shoes, hats and mugs (b-d); (e)-(f) masked descriptors for low-texture toys. Right: the 47-object set.

From Figure 6: Grasping a point clicked in one reference image: the caterpillar's tail (i) and right ear (ii) across ... — From Figure 6: Grasping a point clicked in one reference image: the caterpillar's tail (i) and right ear (ii) across configurations, a class-general point on unseen shoes after training on 4 shoes (iii), and a point on one specific shoe in clutter (iv).

Abstract

What is the right object representation for manipulation? We would like robots to visually perceive scenes and learn an understanding of the objects in them that (i) is task-agnostic and can be used as a building block for a variety of manipulation tasks, (ii) is generally applicable to both rigid and non-rigid objects, (iii) takes advantage of the strong priors provided by 3D vision, and (iv) is entirely learned from self-supervision. This is hard to achieve with previous methods: much recent work in grasping does not extend to grasping specific objects or other tasks, whereas task-specific learning may require many trials to generalize well across object configurations or other tasks. In this paper we present Dense Object Nets, which build on recent developments in self-supervised dense descriptor learning, as a consistent object representation for visual understanding and manipulation. We demonstrate they can be trained quickly (approximately 20 minutes) for a wide variety of previously unseen and potentially non-rigid objects. We additionally present novel contributions to enable multi-object descriptor learning, and show that by modifying our training procedure, we can either acquire descriptors which generalize across classes of objects, or descriptors that are distinct for each object instance. Finally, we demonstrate the novel application of learned dense descriptors to robotic manipulation. We demonstrate grasping of specific points on an object across potentially deformed object configurations, and demonstrate using class general descriptors to transfer specific grasps across objects in a class.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong