TaF-Foundation: Wrench-Grounded Cross-Sensor Tactile Representations
TL;DR They record over 100 million tactile readings paired with measured force/torque from 24 different tactile sensors and train one encoder that maps all of them into a shared, force-grounded feature space. Against the strongest baseline it cuts single-frame force error by 17.1% and raises average real-robot success on four tasks from 23.75% to 46.25%.
What they do
Tactile sensors differ in how they work and what they output (gel camera images, magnetic field vectors, pressure maps), so features learned on one rarely transfer to another. TaF-Foundation uses the measured 6D wrench (force and torque) as a shared physical label: each sensor is paired with wrenches on its own, so no sensor-to-sensor pairing is needed. The frozen encoder then feeds a manipulation policy, TaF-Policy.
How it works
- Data. A Dynamixel-driven rig presses, slides, rotates and rolls more than 200 probe shapes on a sensor while an ATI Axia80 force/torque sensor records the same contact: 101.3M frames and 30,503 episodes from 24 optical, magnetic, pressure and piezoresistive sensors.
- Encoder. Sensor-specific adapters (residual MLPs on image patches, MLPs for vector signals, a conv front end for pressure matrices) produce tokens plus a sensor-identity token. A shared Transformer initialized from DINOv2 then pools them with a learnable query.
- Training. A wrench encoder and decoder learn to reconstruct the wrench. The tactile embedding must predict the wrench through that decoder and match an EMA copy of the wrench embedding.
- TaF-Policy. Frozen features from the left and right tactile sensors join head and wrist camera tokens and robot state in ACT.
Results
- On 21 seen sensors, single-frame force RMSE is 2.321 N vs 2.799 N (AnyTouch 2), and cross-sensor retrieval R@1 is 34.38% vs 21.27%.
- On three held-out sensors it has the best overall few-shot score (0.4468 vs 0.4497 for T3), though HTT is better at the 5% and 10% temporal budgets.
- On a Franka (20 trials per task): 35/60/25/65% on Lift Bottle, Remove Power Bank, Key Insert and Board Clean, vs 20/25/10/40% for AnyTouch 2. Board Clean uses a held-out magnetic sensor. In UniVTAC simulation: 51.33% vs 47.33%.
Limitations
Each new sensor still needs labeled tactile–wrench pairs and a trained input module. Real-world success stays between 25% and 65%: Key Insertion fails on key–keyhole misalignment and Lift Bottle on slippage. Ablations and few-shot results come from a single seed.
Videos
Figures




Abstract
Tactile sensing is essential for contact-rich manipulation, yet heterogeneous sensing mechanisms and observation formats hinder cross-hardware transfer. Since physical contact is governed by Newtonian mechanics, we ground tactile representations in six-dimensional wrenches, establishing a common physical reference across sensing mechanisms. Specifically, we pair tactile observations from all sensors with wrench measurements, enabling shared physical supervision without requiring direct tactile-to-tactile pairing between sensors. To support learning at scale, we develop a dedicated acquisition platform and collect over 100 million paired tactile observations and wrench measurements from 24 sensors spanning four families of sensing principles. Building on this dataset, our framework combines sensor-specific input adaptation with a shared tactile encoder. By learning to reconstruct wrenches, the encoder maps observations from different sensors into a shared, continuous feature space grounded in contact physics. To leverage these representations for manipulation, we introduce TaF-Policy, which integrates tactile features with visual observations and robot states to generate action chunks. We evaluate the learned representations through wrench estimation, cross-sensor retrieval, few-shot adaptation to unseen sensors, and robotic manipulation. On the core benchmark, our method reduces single-frame force RMSE by 17.1% and improves cross-sensor retrieval R@1 by 13.11 percentage points over the strongest baseline. In manipulation experiments, TaF-Policy achieves average success rates of 51.33% across three simulated tasks and 46.25% across four real-world tasks, compared with 47.33% and 23.75%, respectively, for the strongest baselines evaluated on the same task sets. Project page: https://mrjiangjm.github.io/TaF-Foundation/.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
