TipDex-VTLA: Learning When to Refine Actions with Touch for Dexterous Manipulation

TL;DR A vision-tactile-language-action policy for dexterous hands that estimates how far lighting changes are pushing its actions off course and checks incoming touch against its own tactile forecast to decide how strongly to correct actions not yet executed. On four real tasks it succeeds 73.8% of the time under unseen lighting, against 31.3% for the best baseline.
What they do
Lighting changes can shift what a visuomotor policy outputs even when nothing in the scene has physically changed. TipDex-VTLA is a vision-tactile-language-action policy for dexterous hands. It learns how much to trust vision under the current lighting. It also checks whether touch is unfolding as predicted. Together, these two signals set how strongly touch corrects the actions still waiting to be executed.
How it works
- Base policy. A VLM, a tactile encoder (five fingertip and three palm regions) and a flow-matching action expert generate action chunks. A tactile forecaster predicts the next frames of touch. Both are trained first on unaugmented demonstrations, then frozen.
- VASE. Each training image is paired with a relit copy. The frozen policy’s action difference between the two becomes a reliability target, and a small head learns to predict it from the current observation alone, with no paired image needed at deployment.
- TACE. At fast updates, measured touch is compared with the cached forecast for the same time. Low visual reliability or a large tactile mismatch opens a gate that applies bounded corrections to action-expert features, and only unexecuted actions are replaced.
Results
- Two RealMan arms with Inspire hands (364 taxels per hand), 20 trials per task and lighting condition. Average success is 87.5% under standard lighting versus 71.3% for T-Rex, and 73.8% under unseen lighting versus 31.3% for N0-VTLA. ACT drops to 0%.
- Ablations under unseen lighting: removing VASE gives 42.5%, the gate head 40.0%, tactile verification 62.5%.
- UniVTAC simulation (standard lighting only): 84.9% mean over eight tasks versus 83.1% for N0-VTLA.
Limitations
The reliability score is a policy-dependent proxy learned only from illumination perturbations; other visual disturbances are left for future work. In simulation the gain over the best baseline is small.
Figures



Abstract
Changes in illumination can induce action drift in dexterous manipulation even when the physical state remains unchanged. Although existing vision--tactile--language--action (VTLA) policies incorporate tactile feedback, when and how touch refines actions under changing visual conditions remains challenging. We introduce \textbf{TipDex-VTLA}, a VTLA policy that combines estimated visual action reliability and tactile forecast verification to regulate execution-time refinement. Specifically, Visual Action Sensitivity Estimation (VASE) learns visual reliability from a frozen reference policy's action sensitivity to illumination perturbations. Tactile Anticipation and Correction during Execution (TACE) compares predicted and measured touch at matched times and combines the resulting discrepancy with visual reliability to gate bounded feature corrections for unexecuted actions. This effectively helps to refine actions under challenging unseen visual conditions. We evaluate TipDex-VTLA on four real-world dexterous manipulation tasks and the UniVTAC simulation benchmark. TipDex-VTLA trained on data collected under normal visual conditions, achieves average full-task success rates of 87.5% (resp. 73.8%) under normal (resp. challenging unseen) lighting in real-world tasks, significantly outperforming some state-of-the-art methods. We will release the collected data and source code.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
