Deep Visual Foresight for Planning Robot Motion

Chelsea Finn, Sergey Levine

Seminal October 2016

TL;DR Train an action-conditioned video prediction model on 50,000 unlabeled robot pushes, then push objects by choosing actions whose predicted pixel motion carries a user-chosen pixel to its goal. On objects not seen in training it ends 2.52 pixels from the goal on average, against 3.19 for the best calibrated servoing baseline.

What they do

Visual foresight replaces the usual segment-model-simulate pushing pipeline with one learned model of future camera images. The user clicks a pixel on an object and a goal location for it; the robot picks the pushes that its video prediction model says will move that pixel there. Training needs no labels, no camera calibration, no 3D models and no physics simulator.

How it works

Data. 10 7-DoF arms made 50,000 random pushes into bins of 10-20 objects, recording RGB images, gripper pose and commanded pose (3D position plus pitch and yaw).
Model. An action-conditioned convolutional LSTM, taken from prior video prediction work, predicts each next frame as stochastic pixel flow: normalized convolution kernels for several moving parts, combined by learned masks. It is trained only on image reconstruction error.
Planning. The designated pixel starts as a point distribution and is propagated through the predicted flows; a candidate action sequence is scored by the probability that the pixel ends at the goal (log-probabilities summed over several pixels). The cross-entropy method searches actions over a 3-step horizon (about 800 ms) on 64x64 images.
Replanning. New controls about every 200 ms, with optical flow tracking where the designated pixel has moved.

Results

Ten pushing tasks with 7 objects not seen in training. Mean final distance to goal: 2.52 ± 1.06 pixels for visual MPC, versus 5.10 at the start, 4.05 for random actions, 3.79 for servoing the gripper to the goal and 3.19 for servoing along the pixel-to-goal vector with replanning. Both servoing baselines need hand-camera calibration; the model does not.
Specifying opposite motions for an object’s two ends makes the robot rotate it, including objects not in the dataset.

Limitations

Only short-horizon pushing; the horizon and replanning rate are limited by GPU compute. When the arm occludes the object, the tracked pixel lands on the arm and the robot moves the arm instead. Wrong predictions cause failures, such as pushing an empty bottle too close to its center of mass so it translates instead of rotating. The authors say the numbers leave significant room for improvement.

Videos

Figures

From Figure 1: Robots collect unlabeled video of their own pushes, a model of future images is trained on it, and the... — From Figure 1: Robots collect unlabeled video of their own pushes, a model of future images is trained on it, and the robot uses that model to push objects to a goal the user indicates in the image.

From Figure 2: A convolutional LSTM takes the current image, action and gripper state and predicts several stochastic... — From Figure 2: A convolutional LSTM takes the current image, action and gripper state and predicts several stochastic flow kernels plus object masks; compositing them turns the current frame into the predicted next frame. Flows and masks get no direct supervision, only video, actions and states.

From Figure 5: Commanded to rotate an object by moving its two ends in opposite directions (red markers to green-outl... — From Figure 5: Commanded to rotate an object by moving its two ends in opposite directions (red markers to green-outlined goals), the robot pushes it on one side. The arm starts away from the object, so the model has to plan where to make contact. This object was not in the training data.

Abstract

A key challenge in scaling up robot learning to many skills and environments is removing the need for human supervision, so that robots can collect their own data and improve their own performance without being limited by the cost of requesting human feedback. Model-based reinforcement learning holds the promise of enabling an agent to learn to predict the effects of its actions, which could provide flexible predictive models for a wide range of tasks and environments, without detailed human supervision. We develop a method for combining deep action-conditioned video prediction models with model-predictive control that uses entirely unlabeled training data. Our approach does not require a calibrated camera, an instrumented training set-up, nor precise sensing and actuation. Our results show that our method enables a real robot to perform nonprehensile manipulation -- pushing objects -- and can handle novel objects not seen during training.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong