TacPilot: Does Touch Need to Be Everywhere?

TL;DR TacPilot adds touch to a frozen vision-only ACT policy. It re-encodes only the fingertip regions whose readings changed, and its corrections to the arm and gripper apply only when separate gates open. In simulated cylinder uprighting it succeeds in 26 of 30 trials, against 21 for an always-on dense tactile residual and 15 for vision alone.
What they do
Dense tactile residuals re-encode all of the touch signal and correct every action, even when nothing has changed. TacPilot wraps a frozen ACT policy with a tactile residual that decides separately when to refresh tactile features and whether to change the action.
How it works
- Selective encoding. Two 68-taxel, three-axis fingertip pads are split into 32 regions. A lightweight monitor on the raw touch signal triggers a region when its readings jump or drift from their last refresh. Only triggered regions and their graph neighbours are re-encoded; the rest come from a cache.
- Gated corrections. A head proposes bounded Cartesian (translation, rotation) and gripper corrections to the ACT action. Separate pose and gripper gates decide whether each is applied; otherwise ACT’s command passes through unchanged.
- Training. Behavior cloning on expert Cartesian commands, with ACT and the tactile encoder frozen; gate labels mark steps where expert and ACT commands differ. The robot is an OpenArm v2 with Paxini pads, simulated in MuJoCo.
Results
- Simulated cylinder uprighting (two sizes, 30 starts): 26/30 (86.7%), versus 21/30 for a dense always-on tactile residual (a reimplementation of OmniTacTune) and 15/30 for vision-only ACT. On the larger 5×10 cm cylinder: 13/14, versus 8/14 for both baselines.
- Replaying 12 expert trajectories: event-triggered local reuse encodes 56.4% fewer regions and cuts encoding time by 37.3% (30.60 s to 19.17 s), with essentially unchanged action predictions.
- Removing the encoded tactile features raises action error 7.7x, mostly during release.
- On the real robot it performs cylinder uprighting, peg and tube insertion and key turning, shown qualitatively without success rates.
Limitations
The authors call the simulation test an exploratory development evaluation: 19 of the 30 starts were reused from earlier experiments, and material, friction and lighting were fixed. TacPilot also differs from the baselines in training and in a gripper release rule, so the gains cannot be credited to gating alone. Real-robot results are qualitative, and the ablations replay recorded trajectories instead of running new rollouts.
Figures




Abstract
Tactile feedback helps visual manipulation policies respond to local contact, but changes in tactile observations do not always require corrective motion, and sustained corrections do not always require new tactile features. We introduce TacPilot, a tactile residual framework that separates representation updates from action intervention around a frozen visual policy. TacPilot uses a perception gate to determine when tactile features need refreshing and updates only affected regions while reusing unchanged features. A separate intervention gate determines whether bounded Cartesian and gripper residuals modify the nominal visual action. This separation allows tactile perception and action correction to follow different spatial and temporal demands during manipulation. In offline expert-trajectory replay, local feature reuse reduces encoded regions by 56.4% relative to periodic full encoding, with essentially unchanged action predictions. In an exploratory simulation evaluation spanning two cylinder sizes, TacPilot succeeds in 26 of 30 trials (86.7%). We further validate TacPilot on OpenArm v2 equipped with Paxini tactile sensors across four contact-rich manipulation tasks.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
