Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System

Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, Dieter Fox

Seminal October 2019

TL;DR A glove-free teleoperation system that drives a KUKA arm and Allegro hand (23 actuated joints) from four depth cameras watching the operator's bare hand. Two pilots ran 15 dexterous tasks, such as pulling money out of a wallet, and 141 of 150 trials succeeded.

What they do

DexPilot teleoperates a KUKA iiwa7 arm and Allegro hand (23 actuated joints) by watching the operator’s bare hand with four RealSense depth cameras, with no glove or markers. The aim is a low-cost way to collect dexterous demonstrations, including 92 fingertip tactile signals.

How it works

  • Hand tracking. DART fits a 20-joint hand model to the fused point cloud. A two-stage PointNet++ network (hand segmentation plus 23 keypoints) and JointNet (keypoints to joint angles) give it priors that keep it out of wrong local minima.
  • Self-generated labels. DART tracks of the operator wearing a coloured-blob glove labelled 100K bare-hand point clouds (7-8 hours of recording) for those networks.
  • Retargeting. SLSQP minimizes a cost on fingertip and palm vectors, not joint angles. When a finger is near the thumb its target gap becomes 0.1 mm, so pinches close despite tracking error.
  • Control. Riemannian Motion Policies move the palm and avoid the table (arm at 200 Hz, hand at 30 Hz). Latency is about one second.

Results

  • Tracking error is 9.7 mm per keypoint and 1.33 degrees per joint on 7,000 validation images.
  • Two pilots each ran 5 consecutive trials of 15 tasks; 9 of 150 trials failed. In-hand brick rotation was hardest, at 3/5 for both.
  • Mean completion times range from 16 s (brick pick-and-place) to 358 s (opening a container, pilot 1).
  • Peg-in-hole insertion with 0.1 mm clearance succeeds within about 10% of attempts, and only with the board close to the operator.

Limitations

The operator gets no tactile feedback, so precision tasks are hard. The pinch projection interferes with finger gaiting and with releasing small objects. Tracking is less robust for hand sizes and shapes unlike the model, and camera range limits the work volume.

Videos

Full video (YouTube)
Full video (YouTube)

Figures

From Figure 1: DexPilot teleoperating four tasks: righting a Pringles can and placing it in a red bowl (upper left), ...
From Figure 1: DexPilot teleoperating four tasks: righting a Pringles can and placing it in a red bowl (upper left), inserting cups (upper right), picking two cubes at once with four fingers (lower left) and pulling money out of a wallet (lower right).
From Figure 3: The system runs three threads on three computers. The learning thread turns the fused point cloud from...
From Figure 3: The system runs three threads on three computers. The learning thread turns the fused point cloud from four cameras into hand pose and joint angle priors; the tracking thread runs DART hand tracking and kinematic retargeting to the Allegro hand; the control thread runs Riemannian Motion Policies that send joint targets to the KUKA arm and Allegro hand.
From Figure 8: Canonical kinematic retargeting results between the human hand model and the Allegro hand model.
From Figure 8: Canonical kinematic retargeting results between the human hand model and the Allegro hand model.
From Figure 14: Success rate of each teleoperation task for two pilots over five consecutive trials without reset.
From Figure 14: Success rate of each teleoperation task for two pilots over five consecutive trials without reset.
Abstract

Teleoperation offers the possibility of imparting robotic systems with sophisticated reasoning skills, intuition, and creativity to perform tasks. However, current teleoperation solutions for high degree-of-actuation (DoA), multi-fingered robots are generally cost-prohibitive, while low-cost offerings usually provide reduced degrees of control. Herein, a low-cost, vision based teleoperation system, DexPilot, was developed that allows for complete control over the full 23 DoA robotic system by merely observing the bare human hand. DexPilot enables operators to carry out a variety of complex manipulation tasks that go beyond simple pick-and-place operations. This allows for collection of high dimensional, multi-modality, state-action data that can be leveraged in the future to learn sensorimotor policies for challenging manipulation tasks. The system performance was measured through speed and reliability metrics across two human demonstrators on a variety of tasks. The videos of the experiments can be found at https://sites.google.com/view/dex-pilot.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.