3DTouch: A Tactile-aware VLA Model with 3D Geometric Relations for Dexterous Manipulation

TL;DR 3DTouch feeds a pi0 VLA five-finger tactile forces placed in 3D, scene point clouds and explicit hand-object and object-object geometry, all in the robot-base frame. On five real dexterous tasks with a UR7e and XHand it averages 65% success, 20 points above the best baseline and 42 above pi0.
What they do
In dexterous manipulation the fingers hide the contact region from the camera, and scenes that look alike can be in different contact states. 3DTouch gives a pretrained pi0 vision-language-action model three extra 3D inputs: a scene point cloud, the five-finger tactile readings placed where they occur, and explicit hand-object and object-object geometry, all in the robot-base frame.
How it works
- 3D Tactile Grounding. Depth is back-projected to 4,096 scene points and encoded into 33 tokens. Each of the XHand’s 600 taxels (120 per finger) is moved into the base frame by forward kinematics and encoded with its 3D force vector, giving one token per finger.
- Geometric Relation Modeling. Object meshes are reconstructed offline with SAM 3D and placed by a lightweight pose estimator trained on SAM2 and RGBTrack pseudo-labels. Distances and displacements from 17 finger keypoints to nearby surface points give one hand-object token per finger; relations between 8 regions on each of two objects give 5 object-object tokens.
- Conditioning. The tokens enter pi0’s action expert ahead of the noisy actions; the vision-language input and flow-matching loss are unchanged.
Results
- With about 100 demonstrations and 20 real trials per task, 3DTouch averages 65% success, against 45% for Tactile-VLA, 44% for pi0.5 and 23% for pi0.
- Adding raw tactile readings to the robot state is unreliable: it drops pi0.5 on button press from 75% to 30%.
- Ablations: tactile positions in the hand frame instead of the base frame drop the average from 53% to 42%; adding the relation tokens lifts it from 53% to 65%.
- Under shifted or rotated targets, 3DTouch loses 5 points on average, against 13.3 for pi0.5 with tactile.
Limitations
3DTouch needs a reference mesh for each object and a trained pose estimator; the authors name reducing this reliance as future work. It is tested on one arm-hand system with one front camera, and funnel insertion still succeeds only 35% of the time.
Figures



Abstract
In contact-rich dexterous manipulation, occlusion by the fingers and interacting objects obscures local geometry, preventing visual observations from fully capturing contact locations. Visually similar scenes may also differ subtly in the relative positions of interacting objects, leading to different contact states. We propose 3DTouch, a tactile-geometric conditioning approach for vision-language-action (VLA) models. 3DTouch constructs a shared 3D visuo-tactile representation by expressing RGB-D scene points and dense five-finger tactile measurements in the robot-base frame, preserving each taxel's spatial location and force vector. Hand-object and object-object relations derived from hand landmarks and pose-aligned reconstructed surfaces explicitly describe the relative arrangement of interacting entities. Together, these representations condition the VLA action expert on spatially localized contact forces and relative surface geometry, providing explicit cues for contact adjustment and object alignment. We evaluate 3DTouch on five real-world contact-rich dexterous manipulation tasks using a UR7e robot arm equipped with an XHand dexterous hand. 3DTouch improves the average success rate by 42 percentage points over the corresponding VLA baseline and by 20 percentage points over the strongest evaluated alternative. Ablation studies reveal the complementary benefits of scene geometry and spatial tactile feedback, with further gains from shared spatial grounding and explicit geometric relations.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
