Learning Heterogeneous Tactile Representation for General Dexterous Manipulation

Bing Han, Yichuan Peng, Haoyu Dong, Yuanmeng Hu, Yongyi Jia, Fangjing Wang, Kecheng Zheng, Yiming Zhang, Yanhong Zeng, Xiaoqing Zhang, Chen Song, Rundong Zou, Anqi Dong, Yuan Guan, Ka Leong Cheng, Xing Zhu, Yujun Shen, Nan Xue, Wei Ning

November 2026

TL;DR LingBot-Touch is one tactile encoder pretrained on 5,000 hours of piezoresistive, magnetic and visuotactile fingertip data. Its frozen tokens raise the success of four policies on 26 real dual-arm dexterous tasks by 14 to 25 percentage points.

What they do

Tactile sensors differ in physics, layout, resolution and rate, so data from one cannot be pooled with data from another. LingBot-Touch is a tactile encoder trained once on 4,983 hours of fingertip recordings from three sensing principles (piezoresistive and magnetic gloves worn by people, visuotactile Sharpa Wave hands under teleoperation). The frozen encoder is then added to existing policies as extra conditioning tokens.

How it works

Canonical fingertip space. Every taxel gets a 3D position in a shared, unit-radius fingertip frame plus a finger identity and time index. This 5D address enters only through rotary position embeddings, so no fixed taxel count is needed.
Shared backbone. A two-layer MLP tokenizer per sensing principle feeds one Haptic Transformer.
Tactile-JEPA. The model predicts the latent representation of masked taxels from visible ones, with an EMA target encoder, instead of reconstructing the mostly-zero signal.
Use in policies. The 5.46M-parameter Tiny encoder stays frozen; trainable cross-attention readouts produce 40 tokens per hand. It adds under 0.2% to the VLA parameter counts.

Results

Real robots, 26 tasks (20 on Sharpa Wave, 6 on XHand1), 20 trials each: mean success rises from 21.6% to 35.6% for Diffusion Policy, 48.1% to 69.9% for pi-0.5, 48.3% to 73.1% for LingBot-VLA and 45.4% to 68.6% for T-Rex. Raw tactile input gives at most about half that gain.
Frozen-feature probing on Sharpa Wave: fingertip force within 0.81 N and contact center within 0.12 mm, against 2.31 N and 2.19 mm for FTP-1. It also beats FTP-1 on XHand1, whose magnetic sensor model was not in pretraining.
RL in simulation, twelve tasks: 69.0% success, against 45.6% with a CNN trained with the policy and 21.4% without touch.
In the same LingBot-VLA policy on five tasks: 82.2% with LingBot-Touch, 67.5% with the FTP-1 encoder, 55.9% without touch.

Limitations

Experiments cover two hand embodiments. Capacitive sensors are absent from pretraining because the available recordings did not meet quality requirements. The canonical geometry covers only fingertips, so contact on the palm and the rest of the fingers is underrepresented.

Figures

From Figure 1: Top, the 5,000-hour corpus pools human tactile gloves and teleoperated robot hands across piezoresisti... — From Figure 1: Top, the 5,000-hour corpus pools human tactile gloves and teleoperated robot hands across piezoresistive, magnetic and optical sensing; the data are heterogeneous in taxel geometry, resolution and channels, and sparse. Bottom, the frozen encoder recovers contact force, location, extent and dynamics, and its tokens plug into imitation-learning and VLA policies on real dual-arm hands and into reinforcement learning in simulation.

From Figure 3: Signals from magnetic, piezoresistive and visuotactile sensors are mapped to a canonical spherical spa... — From Figure 3: Signals from magnetic, piezoresistive and visuotactile sensors are mapped to a canonical spherical space. Modality-specific tokenizers feed a shared Transformer with 5D rotary position embeddings. The context encoder sees only visible tokens, a predictor fills in the masked ones, and the targets come from an EMA target encoder that sees the full input.

From Figure 11: Mean success over 26 real-world tasks for four policies: the original model, with raw tactile signals... — From Figure 11: Mean success over 26 real-world tasks for four policies: the original model, with raw tactile signals added, and with frozen LingBot-Touch tokens added.

From Figure 14: Stacked cup separation on XHand1. The robot stabilizes the nested stack, grasps and separates the upp... — From Figure 14: Stacked cup separation on XHand1. The robot stabilizes the nested stack, grasps and separates the uppermost cup, and holds it.

Abstract

Tactile sensing is an indispensable modality for contact-rich dexterous manipulation, a capability that hints at the ultimate frontiers of embodied intelligence. However, commercially available tactile sensors are still at a relatively early stage of development, with each type of sensor offering irreplaceable strengths alongside its own drawbacks. A key challenge is thus to free tactile understanding from sensor-specific physics, so that embodied models can effectively leverage heterogeneous tactile data. To this end, we develop LingBot-Touch, a tactile foundation model characterized by the following properties. (i) Our model yields a unified representation across different tactile sensors (i.e., piezoresistive, magnetic, and visuotactile sensors), and is easily extensible to new sensor types. (ii) The resulting tactile tokens are plug-and-play: they can be seamlessly integrated into policy models across paradigms, from diffusion policy to vision-language-action (VLA) models such as $π_{0.5}$ and LingBot-VLA, consistently enhancing manipulation dexterity. (iii) Extensive experiments, spanning simulation and real-robot deployments across multiple hardware platforms, demonstrate that our tactile representation substantially benefits complex manipulation tasks while maintaining strong computational efficiency. We believe the findings in this work demonstrate the potential of leveraging cross-sensor data to scale tactile learning, and offer encouraging insights for the design of future hardware and embodied models.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong