UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policy

Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, Tengyu Liu, Li Yi, He Wang

Seminal March 2023

TL;DR UniDexGrasp samples diverse ShadowHand grasp poses from a tabletop point cloud, then executes one with a goal-conditioned RL policy distilled to point-cloud input. In simulation the full pipeline lifts 58-66% of thousands of objects, including unseen categories, while the best baseline policy reaches 25% even with oracle state.

What they do

UniDexGrasp grasps and lifts arbitrary tabletop objects with a ShadowHand, using only a point cloud and proprioception, in Isaac Gym. Like parallel-gripper pipelines, it works in two stages. First it samples a grasp pose (hand rotation, translation, joint angles) for the observed object. Then a goal-conditioned policy reaches that grasp and lifts the object. The grasp data are 1.12 million synthesized grasps over 5519 objects from 133 categories.

How it works

Grasp proposals. The pose distribution is split into rotation and the rest. GraspIPDF, an implicit density over SO(3), samples a rotation. The point cloud is rotated into that frame and GraspGlow, a conditional normalizing flow, samples translation and joint angles. ContactNet predicts a target contact map, and the pose is optimized for 300 steps to match it without penetration.
Teacher policy. PPO with oracle object state and a reward for reaching the proposed pose. States are expressed relative to the initial hand yaw (SO(2) canonicalization). Objects follow a curriculum: one object, then one category, three categories, and finally all categories.
Student policy. DAgger distills the teacher into a policy that sees only the scene point cloud from five RGB-D cameras plus proprioception.

Results

Proposals reach a Q1 of 0.0423 on seen and 0.0322 on unseen categories, against 0.0269 and 0.0239 for GraspTTA. Their rotation spread is 127.6° against 4.9°. Without the contact optimization, Q1 falls to 0.0013 or lower.
The state-based teacher succeeds on 74% of training objects, 71% of unseen instances and 66% of unseen categories. PPO, DAPG and ILAD reach 14%, 13% and 25% on training objects. Without the curriculum, the teacher reaches 31% on training objects.
The vision student with predicted goals scores 66%, 59% and 58% on the same splits. Goals jittered to 1 cm penetration drop it to 37-47%.

Limitations

Everything runs in simulation, with a floating hand that starts palm-down at a fixed height. The proposal stage uses ground-truth table and object segmentation. Only rigid objects are handled; articulated ones such as scissors are not.

Videos

Figures

From Figure 1: Left, two or three diverse grasp proposals per object that differ in rotation, translation and joint a... — From Figure 1: Left, two or three diverse grasp proposals per object that differ in rotation, translation and joint angles. Right, the goal-conditioned policy grasps the same object in two ways (green and blue trajectories) to match two goal poses.

From Figure 2: Method overview. Stage 1 samples a hand rotation with GraspIPDF, rotates the object point cloud into t... — From Figure 2: Method overview. Stage 1 samples a hand rotation with GraspIPDF, rotates the object point cloud into that frame, samples translation and joint angles with GraspGlow, and refines the pose against a contact map predicted by ContactNet. Stage 2 is the goal-conditioned policy, which takes the goal pose, the scene point cloud and robot proprioception.

From Figure 3: The teacher policy sees object state and a pre-sampled object point cloud and is trained with RL; the ... — From Figure 3: The teacher policy sees object state and a pre-sampled object point cloud and is trained with RL; the student sees only the raw scene point cloud and proprioception and imitates the teacher. Both receive the goal grasp and canonicalized states.

From Figure 4: Eight samples of translation and joint angles for one fixed rotation. The CVAE collapses to a single p... — From Figure 4: Eight samples of translation and joint angles for one fixed rotation. The CVAE collapses to a single pose while GraspGlow gives varied grasps; right, a ground-truth grasp.

Abstract

In this work, we tackle the problem of learning universal robotic dexterous grasping from a point cloud observation under a table-top setting. The goal is to grasp and lift up objects in high-quality and diverse ways and generalize across hundreds of categories and even the unseen. Inspired by successful pipelines used in parallel gripper grasping, we split the task into two stages: 1) grasp proposal (pose) generation and 2) goal-conditioned grasp execution. For the first stage, we propose a novel probabilistic model of grasp pose conditioned on the point cloud observation that factorizes rotation from translation and articulation. Trained on our synthesized large-scale dexterous grasp dataset, this model enables us to sample diverse and high-quality dexterous grasp poses for the object point cloud.For the second stage, we propose to replace the motion planning used in parallel gripper grasping with a goal-conditioned grasp policy, due to the complexity involved in dexterous grasping execution. Note that it is very challenging to learn this highly generalizable grasp policy that only takes realistic inputs without oracle states. We thus propose several important innovations, including state canonicalization, object curriculum, and teacher-student distillation. Integrating the two stages, our final pipeline becomes the first to achieve universal generalization for dexterous grasping, demonstrating an average success rate of more than 60% on thousands of object instances, which significantly outperforms all baselines, meanwhile showing only a minimal generalization gap.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong