Learning Latent Plans from Play

Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, Pierre Sermanet

Seminal March 2019

A human operator collects play data by teleoperating the simulated arm in VR (2.5x speed).

TL;DR Play-LMP learns one goal-conditioned policy from unlabeled, teleoperated play by encoding random windows of play into a latent plan space. On 18 simulated manipulation tasks it reaches 85.5% success from state observations, against 70.3% for 18 task-specific behavioral cloning policies.

What they do

Instead of collecting segmented demonstrations per task, a human teleoperates a simulated 8-DOF arm in VR and plays freely with a desk that has a sliding door, a drawer, three buttons and a block. Play-LMP self-supervises one goal-conditioned policy from these unlabeled logs. For the same 90 minutes of collection, play visits 4.2x more regions of the interaction space than 18 tasks’ worth of expert demonstrations and 14.4x more than random actions.

How it works

Self-labeling. Any window of play shows how the robot got from its first state to its last, so the last frame becomes a reachable goal and the logged actions become labels. The simpler baseline, Play-GCBC, trains a goal-conditioned RNN policy on these windows directly.
Latent plans. Many behaviors connect the same start and goal. Play-LMP is a sequence-to-sequence conditional VAE: a bidirectional RNN encodes the whole window into a latent plan, a plan proposal network predicts a plan distribution from start and goal alone, and a KL term (weight β < 1 to avoid posterior collapse) ties them together. The policy decodes actions from state, goal and plan.
Test time. The proposal network samples a plan from the current state and a goal image. The agent acts at 30 Hz and replans every 32 steps (about 1 Hz).

Results

Average success on 18 tasks from states: Play-LMP 85.5%, Play-GCBC 77.9%, 18 per-task BC policies (1800 demonstrations) 70.3%, multitask BC 66.2%.
From pixels: Play-LMP 69.4%, BC 66.5%, Play-GCBC 58.7%.
Trained on only 30 minutes of play, Play-LMP (71.8%) still beats BC trained on 90 minutes of demonstrations.
Play-trained models degrade less than BC when the starting end-effector position is perturbed by up to 0.4 m, and they retry after failed attempts.

Limitations

All experiments are in simulation, and training and test tasks share one environment; new objects or scenes are not tested. A task is specified only by a goal state, so users cannot say how it should be done. The method assumes play is not heavily skewed toward some objects, and the latent plan distributions are unimodal Gaussians.

Videos

A single Play-LMP policy reaching different goal states, each shown next to its goal.

Play-LMP retrying after failed attempts until it reaches the goal.

Figures

From Figure 9: Play-LMP architecture. A vision network turns 299x299 RGB images into 64 visual features, joined with ... — From Figure 9: Play-LMP architecture. A vision network turns 299x299 RGB images into 64 visual features, joined with 8-d proprioception. A bidirectional-RNN plan recognition network (whole window) and a feedforward plan proposal network (current and goal state) each output a Gaussian over a 256-d latent plan. The RNN policy takes current state, goal and sampled plan and outputs a mixture of discretized logistics over the 8-d action.

From Figure 8: Average success on the 18 tasks (state observations) against minutes of unlabeled play data for one Pl... — From Figure 8: Average success on the 18 tasks (state observations) against minutes of unlabeled play data for one Play-LMP or Play-GCBC policy (left), and against minutes of expert demonstrations for 18 per-task BC policies (right). Play-LMP reaches 85.5% and BC 70.3%; with only 30 minutes of play, Play-LMP reaches 71.8%.

From Figure 4: t-SNE of the learned latent plan space. Task demonstrations colored by type (drawer, button pushing, g... — From Figure 4: t-SNE of the learned latent plan space. Task demonstrations colored by type (drawer, button pushing, grasping, sweeping) form clusters among random play windows (grey), although training used no task labels.

Abstract

Acquiring a diverse repertoire of general-purpose skills remains an open challenge for robotics. In this work, we propose self-supervising control on top of human teleoperated play data as a way to scale up skill learning. Play has two properties that make it attractive compared to conventional task demonstrations. Play is cheap, as it can be collected in large quantities quickly without task segmenting, labeling, or resetting to an initial state. Play is naturally rich, covering ~4x more interaction space than task demonstrations for the same amount of collection time. To learn control from play, we introduce Play-LMP, a self-supervised method that learns to organize play behaviors in a latent space, then reuse them at test time to achieve specific goals. Combining self-supervised control with a diverse play dataset shifts the focus of skill learning from a narrow and discrete set of tasks to the full continuum of behaviors available in an environment. We find that this combination generalizes well empirically---after self-supervising on unlabeled play, our method substantially outperforms individual expert-trained policies on 18 difficult user-specified visual manipulation tasks in a simulated robotic tabletop environment. We additionally find that play-supervised models, unlike their expert-trained counterparts, are more robust to perturbations and exhibit retrying-till-success behaviors. Finally, we find that our agent organizes its latent plan space around functional tasks, despite never being trained with task labels. Videos, code and data are available at learning-from-play.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong