← Paper Digest

TacPilot: Does Touch Need to Be Everywhere?

Yunzhen Zhang, Yuxiao Zhu, Jie Yang, Yufeng Miao, Bing Luo

November 2026

TL;DR TacPilot adds touch to a frozen vision-only ACT policy. It re-encodes only the fingertip regions whose readings changed, and its corrections to the arm and gripper apply only when separate gates open. In simulated cylinder uprighting it succeeds in 26 of 30 trials, against 21 for an always-on dense tactile residual and 15 for vision alone.

What they do

Dense tactile residuals re-encode all of the touch signal and correct every action, even when nothing has changed. TacPilot wraps a frozen ACT policy with a tactile residual that decides separately when to refresh tactile features and whether to change the action.

How it works

  • Selective encoding. Two 68-taxel, three-axis fingertip pads are split into 32 regions. A lightweight monitor on the raw touch signal triggers a region when its readings jump or drift from their last refresh. Only triggered regions and their graph neighbours are re-encoded; the rest come from a cache.
  • Gated corrections. A head proposes bounded Cartesian (translation, rotation) and gripper corrections to the ACT action. Separate pose and gripper gates decide whether each is applied; otherwise ACT’s command passes through unchanged.
  • Training. Behavior cloning on expert Cartesian commands, with ACT and the tactile encoder frozen; gate labels mark steps where expert and ACT commands differ. The robot is an OpenArm v2 with Paxini pads, simulated in MuJoCo.

Results

  • Simulated cylinder uprighting (two sizes, 30 starts): 26/30 (86.7%), versus 21/30 for a dense always-on tactile residual (a reimplementation of OmniTacTune) and 15/30 for vision-only ACT. On the larger 5×10 cm cylinder: 13/14, versus 8/14 for both baselines.
  • Replaying 12 expert trajectories: event-triggered local reuse encodes 56.4% fewer regions and cuts encoding time by 37.3% (30.60 s to 19.17 s), with essentially unchanged action predictions.
  • Removing the encoded tactile features raises action error 7.7x, mostly during release.
  • On the real robot it performs cylinder uprighting, peg and tube insertion and key turning, shown qualitatively without success rates.

Limitations

The authors call the simulation test an exploratory development evaluation: 19 of the 30 starts were reused from earlier experiments, and material, friction and lighting were fixed. TacPilot also differs from the baselines in training and in a gripper release rule, so the gains cannot be credited to gating alone. Real-robot results are qualitative, and the ablations replay recorded trajectories instead of running new rollouts.

Figures

From Figure 1: Touch changes are local and intermittent. TacPilot pairs event-triggered local tactile encoding with g...
From Figure 1: Touch changes are local and intermittent. TacPilot pairs event-triggered local tactile encoding with gated, bounded task-space corrections; the photos show the four physical tasks (upright, peg, tube, key).
From Figure 2: The TacPilot framework. Behavior cloning trains the residual and gating head while ACT stays frozen. A...
From Figure 2: The TacPilot framework. Behavior cloning trains the residual and gating head while ACT stays frozen. At inference a change detector triggers local tactile feature updates that reuse cached features, bounded residuals correct the nominal ACT action, and independent pose and gripper gates decide whether each correction is applied.
From Figure 3: Simulated cylinder uprighting from the same start for (A) vision-only ACT, (B) the dense tactile resid...
From Figure 3: Simulated cylinder uprighting from the same start for (A) vision-only ACT, (B) the dense tactile residual and (C) TacPilot. ACT still holds the cylinder at the end and the dense residual releases too early so it topples; both time out, while TacPilot places, releases and leaves it standing.
From Figure 5: Four contact-rich tasks on the real OpenArm v2: (A) cylinder uprighting, (B) solid-peg insertion, (C) ...
From Figure 5: Four contact-rich tasks on the real OpenArm v2: (A) cylinder uprighting, (B) solid-peg insertion, (C) hollow-tube insertion and (D) key manipulation, each with external, head and wrist camera views and left-pad tactile readings.
Abstract

Tactile feedback helps visual manipulation policies respond to local contact, but changes in tactile observations do not always require corrective motion, and sustained corrections do not always require new tactile features. We introduce TacPilot, a tactile residual framework that separates representation updates from action intervention around a frozen visual policy. TacPilot uses a perception gate to determine when tactile features need refreshing and updates only affected regions while reusing unchanged features. A separate intervention gate determines whether bounded Cartesian and gripper residuals modify the nominal visual action. This separation allows tactile perception and action correction to follow different spatial and temporal demands during manipulation. In offline expert-trajectory replay, local feature reuse reduces encoded regions by 56.4% relative to periodic full encoding, with essentially unchanged action predictions. In an exploratory simulation evaluation spanning two cylinder sizes, TacPilot succeeds in 26 of 30 trials (86.7%). We further validate TacPilot on OpenArm v2 equipped with Paxini tactile sensors across four contact-rich manipulation tasks.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.