Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

DexYCB: A Benchmark for Capturing Hand Grasping of Objects

Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, Dieter Fox

Seminal April 2021

Ground-truth hand and object poses rendered on all 8 synchronized camera views.

TL;DR A marker-less dataset of 10 people grasping 20 YCB objects: 582K RGB-D frames from 8 cameras, with 3D hand and object poses fit to crowdsourced keypoints. Hand-pose models trained on it transfer to HO-3D better than the reverse (1.4-1.9x versus 3.4-3.7x error increase), and it adds a benchmark for safe human-to-robot handover grasps.

What they do

Joint hand and object pose estimation needs real data of hands handling objects, but gloves, magnetic sensors or mocap markers can change how hands move and look. DexYCB records 10 subjects picking up 20 YCB-Video objects from a table without markers, and labels the 3D hand (MANO) and 6D object poses. The paper benchmarks existing methods on 2D detection, 6D object pose and 3D hand pose, and adds a robot task: generating handover grasps that do not pinch the person’s hand.

How it works

  • Capture. 8 calibrated, synchronized RealSense D415 cameras record color and depth at 640x480 and 30 fps. In each 3 s trial the subject picks up a target object placed among 2-4 others and holds it up: 1,000 sequences, 582K frames.
  • Annotation. Mechanical Turk workers track 21 hand joints, plus 2 landmarks of their choosing on each object, in every view.
  • Pose solving. Per frame, MANO and object poses are optimized against a depth term (signed distance of the merged point cloud to the meshes), keypoint reprojection and a hand-pose prior, initialized from the previous frame.
  • Splits. Default, unseen subjects (S1), unseen views (S2) and unseen grasped objects (S3).

Results

  • Cross-dataset hand pose: tested on HO-3D, a DexYCB-trained model has 1.4-1.9x the error of an HO-3D-trained one (31.76 vs 18.05 mm root-relative); tested on DexYCB, an HO-3D-trained model has 3.4-3.7x the error (48.30 vs 12.97 mm).
  • 6D object pose: grasping hurts, PoseCNN AR drops from 56.53 on static objects to 41.65 on grasped ones. On unseen subjects, DeepIM RGB-D (57.54) and CosyPose (57.43) lead.
  • 3D hand pose from RGB: 52.26 mm absolute error but 6.83 mm after Procrustes alignment; unseen views are hardest (80.63 mm).
  • Handover: better object pose gives better grasps, and CosyPose beats DeepIM RGB-D on grasping despite near-equal AR.

Limitations

Lab capture on a black background: a DexYCB-trained hand detector makes false positives, misses and bad keypoints on COCO images. One task only (pick up and hold), with subjects always facing the same way. The handover baseline misses hands occluded by the object, and it is scored offline against pre-generated parallel-jaw grasps.

Videos

Full video (YouTube)

Figures

From Figure 1: One camera view of a DexYCB capture, with the ground-truth 3D hand and object poses rendered on the da...
From Figure 1: One camera view of a DexYCB capture, with the ground-truth 3D hand and object poses rendered on the darkened image.
From Figure 2: The capture rig, 8 RGB-D cameras (red circles) around a tabletop of YCB objects.
From Figure 2: The capture rig, 8 RGB-D cameras (red circles) around a tabletop of YCB objects.
From Figure 4: Precision-coverage curves for handover grasp generation on S1 (unseen subjects), one curve per object ...
From Figure 4: Precision-coverage curves for handover grasp generation on S1 (unseen subjects), one curve per object pose estimator.
From Figure 5: Grasps predicted from the estimated object pose (textured model) and hand segmentation (blue mask). Gr...
From Figure 5: Grasps predicted from the estimated object pose (textured model) and hand segmentation (blue mask). Green grasps cover a successful reference grasp; red ones collide with the object or hand.
Abstract

We introduce DexYCB, a new dataset for capturing hand grasping of objects. We first compare DexYCB with a related one through cross-dataset evaluation. We then present a thorough benchmark of state-of-the-art approaches on three relevant tasks: 2D object and keypoint detection, 6D object pose estimation, and 3D hand pose estimation. Finally, we evaluate a new robotics-relevant task: generating safe robot grasps in human-to-robot object handover. Dataset and code are available at https://dex-ycb.github.io.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.