Which Physical Tactile Cues Matter for Contact-Rich Imitation Learning? A Controlled Real-Robot Study

Xuhao Qin, Xiyan Huang, Haoyu Dong, Xi Zhang, Chenxi Xiao

November 2026

Test-tube reorientation and insertion, seen from above.

TL;DR From one GelSight-style sensor stream they derive tactile RGB, normal/depth maps and marker flow, and compare them as Diffusion Policy inputs over 720 real-robot trials. Touch raises success from 57.8% to 78.9%, and marker flow is the best single cue at 74.4%.

What they do

Vision-based tactile sensors carry several physical signals in one image stream, but policies usually treat that stream as a plain RGB image. This paper asks which signal actually helps imitation learning. From one sensor they derive three cue representations, tactile appearance, contact geometry and tangential deformation, and compare them on three real tasks with the same demonstrations and the same Diffusion Policy backbone.

How it works

Platform. A LeRobot-compatible parallel-jaw gripper with a GelSight Wedge-style sensor (marker-embedded elastomer, internal camera, calibrated lighting) on one fingertip, plus a global and a wrist camera.
Cues. Tactile RGB is the raw image. A calibration network predicts a surface-normal map, which Poisson reconstruction turns into depth. Tracked marker displacements are interpolated into a dense flow field. “Full tactile” encodes all three with separate frontends.
Studies. First the tactile input is varied against a vision-only baseline. Then, with full tactile fixed, ResNet-18 is compared with a tactile-pretrained Sparsh-DINO encoder, and concatenation with cross-attention fusion. Each configuration gets 30 trials per task.

Results

Full tactile succeeds in 71/90 trials versus 52/90 for vision only. Gains are 30.0 points on test-tube reorientation, 23.3 on wiping and 10.0 on peg insertion.
Marker flow is the best single cue (67/90), ahead of tactile RGB (66/90) and normal/depth (60/90); full tactile beats the best single cue by one success per task.
Cross-attention raises full-tactile success to 77/90 with ResNet-18 and 79/90 with Sparsh-DINO, so ResNet-18 with cross-attention roughly matches Sparsh-DINO with concatenation (75/90).
Failure analysis suggests normal/depth helps establish contact, while marker flow helps regulate contact once it exists.

Limitations

No direct force measurements, frontend architectures differ between cue configurations, and the study uses one tactile sensor, a parallel-jaw gripper and three tabletop tasks.

Videos

Blackboard wiping with sustained surface contact.

Peg-in-hole insertion under tight clearance.

Figures

From Figure 1: A LeRobot-compatible visuo-tactile platform converts the same tactile observation into tactile RGB, no... — From Figure 1: A LeRobot-compatible visuo-tactile platform converts the same tactile observation into tactile RGB, normal/depth maps and marker-flow fields, which are compared as policy inputs, along with tactile encoders and visuo-tactile fusion strategies, on three real-robot tasks.

From Figure 2: The visuo-tactile system and Diffusion Policy pipeline. Tactile RGB and geometry use image encoders, m... — From Figure 2: The visuo-tactile system and Diffusion Policy pipeline. Tactile RGB and geometry use image encoders, marker flow uses an MLP, and wrist-camera and tactile-RGB features can optionally interact through cross-attention before fusion and action prediction.

From Figure 5: Successful executions of the three tasks over time (left) and typical failures under tactile RGB, norm... — From Figure 5: Successful executions of the three tasks over time (left) and typical failures under tactile RGB, normal/depth, marker-flow and full-tactile inputs (right).

Abstract

Vision-based tactile sensors capture multiple aspects of physical interaction within a single image stream, including contact appearance, geometry, indentation depth, and tangential deformation. However, existing visuo-tactile policies often directly process tactile observations as images, making it difficult to determine which physical cues contribute to policy learning. This paper presents a controlled real-robot study that compares physically interpretable tactile signals using the LeRobot platform, enhanced with our visuo-tactile perception system. From the same tactile stream, we derive tactile RGB images, normal/depth maps, and marker-flow fields, representing contact appearance, geometry, and tangential deformation, respectively. Using identical task demonstrations and the same policy backbone, we evaluate the individual and combined contributions of these tactile cues. We further compare tactile encoders and visuo-tactile fusion strategies. Evaluation across 720 real-robot trials yields four key findings: (1) tactile perception substantially improves the overall success rate from 57.8% to 78.9%, although its contribution varies across task stages; (2) marker flow is the most effective individual cue, achieving a success rate of 74.4% and playing a particularly important role in regulating established contact; (3) cue-factorized encoding consistently improves performance, highlighting that different physical cues require tailored processing techniques; and (4) although the effectiveness of tactile fusion is task-dependent, cross-attention offers a lightweight solution with performance comparable to that of pretrained tactile encoders.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong