Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

One-Shot Visual Imitation Learning via Meta-Learning

Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, Sergey Levine

Seminal September 2017

TL;DR Meta-train a vision-based policy with MAML so that one gradient step on a single demonstration adapts it to a new task. A real PR2 places held items onto unseen containers with 90% one-shot success, against 25% for LSTM and contextual baselines.

What they do

Learning a visuomotor skill from raw pixels usually takes many demonstrations per task. Meta-imitation learning (MIL) meta-trains a policy on demonstrations of many other tasks so that a gradient update on one demonstration of a new task, with new goals or unseen objects, yields a policy for that task.

How it works

  • MAML for imitation. Each meta-training step samples a batch of tasks and two demonstrations per task. One adapts the parameters with a behavioral-cloning (mean squared error) gradient step; the adapted policy’s error on the other is backpropagated to the initial parameters.
  • Policy. Strided convolutions, a spatial soft-argmax into feature points, concatenation with the robot configuration, then fully connected layers. Layer normalization replaces batch normalization, which was not effective on time-correlated demonstration frames.
  • Bias transformation. A learned vector concatenated to a hidden layer gives the gradient more direct control over the bias; the authors found it made meta-learning significantly more stable.
  • Two heads. Separate pre- and post-update final layers amount to a meta-learned inner loss. That loss need not use actions, so the policy can adapt from a demonstration video alone (meta-training still uses expert actions).

Results

  • Simulated pushing (7-DoF torque control, held-out meshes and textures): 85.8% one-shot success with the largest dataset, on average 6.5% above the LSTM baseline across dataset sizes. From video only, MIL reaches 66.4% against 34.2% for LSTM.
  • Real PR2 placing, meta-trained on about 1300 teleoperated demonstrations and tested on 29 held-out objects: 90% success against 25% for LSTM and contextual policies, and 68.3% from video only.

Limitations

The paper does not address compounding errors in imitation. Task diversity is limited apart from object diversity. Learning from video alone costs accuracy, and meta-training still needs actions. On the robot, the final gripper opening is hand-coded.

Figures

From Figure 1: The robot learns to place a new object into a new container from a single demonstration.
From Figure 1: The robot learns to place a new object into a new container from a single demonstration.
From Figure 2: The policy architecture with a bias transformation and two heads. Green arrows and boxes mark weights ...
From Figure 2: The policy architecture with a bias transformation and two heads. Green arrows and boxes mark weights that are part of the meta-learned policy parameters.
From Figure 4: One-shot simulated pushing success on held-out tasks versus meta-learning dataset size. MIL beats the ...
From Figure 4: One-shot simulated pushing success on held-out tasks versus meta-learning dataset size. MIL beats the contextual (goal image) and LSTM (demonstration) policies; a random policy gets 0.45%.
Abstract

In order for a robot to be a generalist that can perform a wide range of jobs, it must be able to acquire a wide variety of skills quickly and efficiently in complex unstructured environments. High-capacity models such as deep neural networks can enable a robot to represent complex skills, but learning each skill from scratch then becomes infeasible. In this work, we present a meta-imitation learning method that enables a robot to learn how to learn more efficiently, allowing it to acquire new skills from just a single demonstration. Unlike prior methods for one-shot imitation, our method can scale to raw pixel inputs and requires data from significantly fewer prior tasks for effective learning of new skills. Our experiments on both simulated and real robot platforms demonstrate the ability to learn new tasks, end-to-end, from a single visual demonstration.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.