Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

BAT-WAM: Bridging Action and Touch in World-Action Models for Dexterous Hands

Dong Fang, Feixuan Huang, Yuanxin Zhong, Yuxiang Gao, Qi Song, Yunlong Wang, Xiaosong Jia, Yu-Gang Jiang

November 2026

TL;DR BAT-WAM tokenizes whole-hand touch and joint state separately and links them through an RGB-conditioned Transformer, so each can be reconstructed from the other before feeding a world-action model. On seven real tasks with a tactile-gloved dexterous hand it averages 79.3% success versus 66.4% for the baseline.

What they do

World-action models for dexterous hands need the hand configuration and its contacts, but joint-state and whole-hand tactile readings can be partly or wholly missing, and RGB alone does not recover them. BAT-WAM builds on the observation that hand state and touch are correlated. It learns a shared interface in which touch and joint state supervise each other under RGB context, and passes the result as extra input tokens to a pi0.5 backbone.

How it works

  • Separate tokenizers. Touch (a spatial pressure map) and joint state each get their own encoder, codebook and decoder: 196 touch tokens and 8 joint tokens per hand.
  • RGB as context. A frozen SigLIP encoder with learned queries gives continuous image tokens that are neither quantized nor reconstructed.
  • Bridge Transformer. All three streams are mapped to a common 256-dimensional space. Training masks one modality and reconstructs it from the others, with an attention mask hiding the unavailable inputs.
  • Temporal extension. Past touch and joint states plus RGB up to the target time predict the touch map at the target time.

Results

  • HT-Bench, touch fully masked: RMSE 0.0145 versus 0.0221 for image-conditioned HandTouch, a 34.73% reduction. With local masks, hidden-region RMSE is 0.0137 versus 0.0224.
  • Future touch prediction: RMSE 0.0157 versus 0.0211 for HandTouch.
  • Real robot (AgileX PiPER arm, Linker Hand L6 with a full-hand tactile glove, 200 demonstrations per task): mean success over seven tasks is 79.3%, against 66.4% for OpenWAM-alpha and 69.3% for OpenWAM-alpha with HandTouch. Whiteboard wiping reaches 45% versus 15% and 25%.
  • Unseen colors, sizes, shapes, layouts and poses: 54.5% average versus 38.5% and 42.0%.

Limitations

The paper has no limitations section. It notes that changes in object geometry remain hard: success is 37.5% on unseen sizes and 27.5% on unseen shapes, against 87.5% and 72.5% in the seen settings. All tasks share one tabletop setup, and Close Drawer is the one task where BAT-WAM is not the best method.

Figures

From Figure 1: Overview. BAT-WAM bridges state and contact information with RGB observation.
From Figure 1: Overview. BAT-WAM bridges state and contact information with RGB observation.
From Figure 2: Pipeline of BAT-WAM. A shared bridge Transformer integrates separately tokenized touch and joint state...
From Figure 2: Pipeline of BAT-WAM. A shared bridge Transformer integrates separately tokenized touch and joint states with continuous RGB features. Masked cross-modal reconstruction and temporal prediction train the representation interface, whose outputs, together with language instructions, condition a WAM backbone for action generation.
From Figure 3: Under HTBench, BAT-WAM is the best on both displayed metrics for the Action+Image condition.
From Figure 3: Under HTBench, BAT-WAM is the best on both displayed metrics for the Action+Image condition.
From Figure 4: Real-world experiment environment and representative task examples. The left column shows two views of...
From Figure 4: Real-world experiment environment and representative task examples. The left column shows two views of the shared tabletop setup.
Abstract

Dexterous manipulation with world action models (WAMs) relies on hand configuration and contact information, yet joint-state and whole-hand tactile observations may be partially or entirely missing. RGB images provide scene context and hand-object geometry but are insufficient to reliably recover these missing modalities. We further observe a correlation between hand state and tactile information. Motivated by this observation, we introduce BAT-WAM, a framework that bridges visual, joint-state, and tactile representations for WAMs. BAT-WAM learns separate codebooks and decoders for the tactile and action streams. An RGB-conditioned continuous latent bridge connects the two streams for bidirectional learning, while visual information serves as continuous context. Under the HTBench evaluation protocol, BAT-WAM reduces the root mean squared error (RMSE) of tactile reconstruction by 34.73% under fully masked conditions and 38.84% under partially masked conditions, relative to an image-conditioned HandTouch checkpoint. For future tactile prediction conditioned on target-time RGB and historical actions, it reduces mean RMSE by 85.56% relative to OpenWAM with HandTouch. Real-robot experiments further show that BAT-WAM consistently outperforms related WAM baselines in dexterous manipulation. Generalization experiments on unseen objects and scenes also demonstrate substantial improvements over baseline models. These findings support joint modeling of visual context, hand state, and touch as a representation approach for multimodal WAMs.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.