Manipulation by Feel: Touch-Based Control with Deep Predictive Models

Stephen Tian, Frederik Ebert, Dinesh Jayaraman, Mayur Mudigonda, Chelsea Finn, Roberto Calandra, Sergey Levine

Seminal March 2019

TL;DR A video-prediction model learned from random pokes predicts future GelSight tactile images, and MPC uses it to push a ball, analog stick or 20-sided die until the sensor reading matches a goal tactile image. On die rolling it reaches the target face in 26 of 30 trials, against 14 of 30 for a hand-designed centroid controller.

What they do

Deep tactile MPC controls a robot by touch alone. A 4 cm GelSight-style sensor on a 3-axis CNC stage must move an object until the sensor reads a user-given goal tactile image. The three tasks are rolling a ball bearing to a spot on the sensor, deflecting an analog stick, and rolling a 20-sided die to a chosen face. The object is hidden under the finger, so a remote camera cannot see it. No rewards or manual labels are used: the model learns only from random autonomous interaction.

How it works

Data. Random moves along the three axes, with steps of up to ±6 mm, give 7400 ball, about 3000 stick and 4500 die trajectories of 15-18 steps each. Resets are passive: a curved surface for the ball, the stick’s springs, and a motor-wound thread for the die.
Model. An action-conditioned recurrent video-prediction network, taken from prior visual-foresight work, predicts future 48x64 tactile images from the current image and candidate actions.
Planning. The cross-entropy method (3 iterations of 100 samples) minimizes the pixel MSE between predicted frames and the goal image. Each action is repeated 3 times, and the plan is recomputed every step. Control runs at about 1 Hz.

Results

Die rolling reaches the goal face in 26/30 trials (86.6%), against 14/30 (46.6%) for a baseline that moves in a straight line toward the goal contact centroid.
Median distance between the final and goal pressure centroids (hand-annotated, Table I in mm): 2.10 vs 2.97 for the ball and 5.31 vs 8.86 for the analog stick.
On the centroid-distance and image-MSE threshold curves (30 ball and 15 stick trials), MPC beats the baseline for both tasks. On the stick, the planner lifts off, repositions and comes back down to make the goal indentation.

Limitations

Control is short-horizon, tens of steps, so tasks with multiple objects or repeated finger gaits are out of reach. A single finger limits behavior to simple rearrangement. Pixel MSE to the goal image does not necessarily reflect the true object position, so evaluation relies on hand annotation. Data collection needed task-specific reset mechanisms.

Videos

Figures

From Figure 3: Deep tactile MPC. The learned model predicts tactile images for sampled action sequences, each sequenc... — From Figure 3: Deep tactile MPC. The learned model predicts tactile images for sampled action sequences, each sequence is scored by its distance to the goal tactile image, and the first action of the best one is executed.

From Figure 5: Ball-bearing rollout. Top: goal tactile image; middle: predicted frames for the lowest-cost action seq... — From Figure 5: Ball-bearing rollout. Top: goal tactile image; middle: predicted frames for the lowest-cost action sequence; bottom: the tactile and side images of the trajectory the robot actually took.

From Figure 9: A successful die roll from face 20 to face 8, with the predicted frames for the best action sequence (... — From Figure 9: A successful die roll from face 20 to face 8, with the predicted frames for the best action sequence (second row) and the real trajectory below.

From Figure 7: Ball task. Number of the 30 test trajectories whose final pressure centroid lies within a given pixel ... — From Figure 7: Ball task. Number of the 30 test trajectories whose final pressure centroid lies within a given pixel distance of the goal (human-annotated), for tactile MPC (orange) and the hand-designed baseline (blue).

Abstract

Touch sensing is widely acknowledged to be important for dexterous robotic manipulation, but exploiting tactile sensing for continuous, non-prehensile manipulation is challenging. General purpose control techniques that are able to effectively leverage tactile sensing as well as accurate physics models of contacts and forces remain largely elusive, and it is unclear how to even specify a desired behavior in terms of tactile percepts. In this paper, we take a step towards addressing these issues by combining high-resolution tactile sensing with data-driven modeling using deep neural network dynamics models. We propose deep tactile MPC, a framework for learning to perform tactile servoing from raw tactile sensor inputs, without manual supervision. We show that this method enables a robot equipped with a GelSight-style tactile sensor to manipulate a ball, analog stick, and 20-sided die, learning from unsupervised autonomous interaction and then using the learned tactile predictive model to reposition each object to user-specified configurations, indicated by a goal tactile reading. Videos, visualizations and the code are available here: https://sites.google.com/view/deeptactilempc

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong