Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Visuo-tactile Diffusion Policy for Robotic Manipulation with Auxiliary Tactile Dynamics Prediction

Kaihong Ouyang, Zihao Su, Chengxuan Zheng, Weiyi Lai, Dong Zhou

November 2026

TL;DR A visuotactile diffusion policy for simulated UR5 assembly, with a training-only head that predicts how the fingertip images change two steps ahead. The full model reaches 59.0% success against 42.7% for vision only, while adding touch without the auxiliary loss drops to 33.7%.

What they do

A diffusion policy for a simulated UR5 assembly task (grasp an object, transport it to a target region, align it) that conditions on two camera views and two fingertip “tactile” images. The main addition is a training-only auxiliary head that predicts how the fingertip images will change over the next two actions. The fingertip images are rendered geometry proxies in MuJoCo, not readings from a real tactile sensor.

How it works

  • Encoders. Two ImageNet-pretrained ResNet-18s with spatial softmax handle the scene and wrist cameras; they are frozen after training a vision-only policy first. A three-layer CNN encodes the left and right fingertip images stacked as 6 channels.
  • Temporal fusion. A GRU aggregates four frames of visual and tactile features. Its output joins the flattened visual history as the condition for a FiLM-conditioned 1D U-Net, which denoises a 16-step action tensor (8 actions executed) in 10 DDIM steps.
  • Auxiliary tactile dynamics. From the GRU state and the next two demonstrated actions, an MLP predicts the 4x4-pooled change in the fingertip images two steps ahead (Smooth L1 loss, weight 0.05). The head is dropped at inference.
  • Data. Expert demonstrations, corrections from a scripted teacher with full simulator state, and the robot’s own rollouts, sampled at 32.7%, 50.0% and 17.3%.

Results

  • The full model reaches 59.0% ± 2.6% success over three seeds (100 matched episodes each), against 42.7% ± 1.5% for vision only: a 16.3-point gain (18, 15 and 16 points per seed).
  • Adding touch without the auxiliary loss hurts: 33.7% ± 0.6%.
  • The auxiliary loss with zeroed tactile input gives no gain: 42.3% ± 1.5%.

Limitations

Simulation only, and the tactile proxies are cleaner than real sensor images and do not measure force. One task setup and three seeds (sign test p = 0.25). Real tactile sensors and a physical robot are left to future work.

Figures

From Figure 1: (a) ResNet-18 encoders for the eye-in-hand and eye-to-hand views and a CNN for the two fingertip tacti...
From Figure 1: (a) ResNet-18 encoders for the eye-in-hand and eye-to-hand views and a CNN for the two fingertip tactile images; (b) a GRU fuses four frames of visual and tactile features, and its output is joined with the visual history; (c) a conditional U-Net generates the action sequence by DDIM denoising; (d) a training-only head predicts the tactile change two steps ahead.
From Figure 2: Eye-to-hand and eye-in-hand camera images (top) and the left and right fingertip tactile-proxy images ...
From Figure 2: Eye-to-hand and eye-in-hand camera images (top) and the left and right fingertip tactile-proxy images (bottom) at the same time step. The fingertip colors encode local geometry and interaction, not calibrated contact forces.
Abstract

Long-horizon robotic assembly requires reliable perception and action across grasping, alignment, and insertion. Vision captures global object pose and scene context, where as fingertip touch provides local contact cues that are especially useful as contact states change. We present a visuotactile diffusion policy that combines four-step histories from two camera views and two fingertip tactile images to condition action-sequence generation. A recurrent temporal representation integrates recent observations from both modalities. To encourage it to capture how contact evolves under action, an auxiliary objective predicts fingertip-image changes two steps ahead from the temporal representation and demonstrated actions. This objective is used only during training and adds no inference-time computation. Across three independent training seeds in MuJoCo assembly experiments, the full model achieves an average success rate of $59.0\% \pm 2.6\%$, improving on the visual-only baseline ($42.7\% \pm 1.5\%$) by 16.3 percentage points.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.