Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation

Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, Pieter Abbeel

Seminal October 2017

TL;DR A consumer VR headset and hand controllers are used to teleoperate a PR2, and behavioral cloning on the recorded demos learns pixel-to-action policies for ten real manipulation tasks. With under 30 minutes of demonstrations per task, success rates range from 80% to 98.9%.

What they do

Collecting demonstrations for learning visuomotor policies is hard: in kinesthetic teaching the human’s arms end up in the camera image, and other teleoperation setups can be expensive. The authors build a teleoperation system for a PR2 from a $600 Vive VR kit, then train deep policies from raw RGB-D pixels by behavioral cloning. They use a single architecture and one set of hyperparameters for all ten real-world tasks.

How it works

  • VR interface. The colored point cloud from an RGB-D camera on the robot’s head is rendered in Unity, and the virtual view follows the operator’s head instantly, avoiding the motion sickness of streamed stereo images. Operator and robot share the same observations.
  • Control. The hand controller’s pose is the gripper’s target pose, sent at 10 Hz to a Jacobian-transpose torque controller. When the gripper is blocked, the gap between target and actual pose scales the applied force, so operators can push and insert.
  • Policy. RGB and depth images plus five steps of end-effector points go into a CNN with a spatial soft-argmax. The network outputs right-arm linear and angular velocity and a gripper open/close command.
  • Losses. l1, l2 and a direction-alignment loss on the velocity, cross-entropy for the gripper, and auxiliary losses that predict current and final gripper pose from the visual features.

Results

  • Test success rates: 80% (grasp-place-x2) to 98.9% (pushing), using 10.1-25.0 min of demos per task.
  • Nail task: 88.9% with 193 demos, 77.8% with 115, 50% with 67 (about 4.2 min).
  • Grasp-and-place with vs. without auxiliary losses: 96% vs. 80% (109 demos), 53% vs. 26% (55), 28% vs. 20% (11).
  • The nail policy generalized to a nail position 3.5 cm from the one used in every demo.

Limitations

Each task gets its own policy, and only the right arm is controlled. Learned policies are often slower and less direct than the demos, sometimes pause, or retry grasps. Failure modes include knocking over the bottle, stalling within 3 mm of the cube slot, and missing the second fruit. The authors did not search for the minimum number of demos needed.

Figures

From Figure 1: an operator in a VR headset teleoperates the PR2 with hand-tracked controllers.
From Figure 1: an operator in a VR headset teleoperates the PR2 with hand-tracked controllers.
From Figure 2: the operator's first-person view inside the VR system, with overlaid visualizations to guide the demon...
From Figure 2: the operator's first-person view inside the VR system, with overlaid visualizations to guide the demonstration.
From Figure 3: the policy network. RGB and depth images pass through convolutions and a spatial softmax, which feeds ...
From Figure 3: the policy network. RGB and depth images pass through convolutions and a spatial softmax, which feeds auxiliary predictions and, with recent end-effector positions, the fully connected layers that output the control.
From Figure 4: successful trials of the learned policies on all ten tasks, showing the camera input at the start, mid...
From Figure 4: successful trials of the learned policies on all ten tasks, showing the camera input at the start, middle and end of each trial.
Abstract

Imitation learning is a powerful paradigm for robot skill acquisition. However, obtaining demonstrations suitable for learning a policy that maps from raw pixels to actions can be challenging. In this paper we describe how consumer-grade Virtual Reality headsets and hand tracking hardware can be used to naturally teleoperate robots to perform complex tasks. We also describe how imitation learning can learn deep neural network policies (mapping from pixels to actions) that can acquire the demonstrated skills. Our experiments showcase the effectiveness of our approach for learning visuomotor skills.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.