MDF++: Visuo-Tactile World Action Model with End-to-end Representation Learning

Zixuan Huang, Jinhao Yu, Zachary Charlick, Dmitry Berenson

November 2026

Uncut bolt-screwing evaluation on a car engine, with stage labels and a running success count.

TL;DR A visuo-tactile world-action model that trains its camera and GelSight encoders together with a diffusion transformer, updating each encoder only through the losses for predicting the other modalities and actions. On a real car engine it screws in bolts 16/20 times, versus 10/20 for the video-based UWM.

What they do

Threading a bolt or nut depends on contact cues that a camera alone misses. World-action models (WAMs), which predict future observations along with actions, usually work in the latent space of a frozen, reconstruction-trained image encoder. MDF++ extends Multimodal Diffusion Forcing to vision, touch (GelSight Mini), force and proprioception, and trains the encoders together with the diffusion transformer. Done naively, this collapses the features into latents that are easy to denoise but weak for control.

How it works

Per-token noise. Each modality at each timestep gets its own noise level, so one transformer can act as a policy, a world-action model, or a forward or inverse dynamics model depending on which tokens stay clean. Training samples these four patterns and random masking with equal probability.
Selective gradient flow. Each encoder (ViT-Tiny for RGB and tactile images) is updated only by the losses for predicting the other streams and the actions, never by its own denoising loss.
Real-robot data. 300 bolt-screwing and 255 nut-mating demonstrations from a UMI handheld gripper fitted with GelSight sensors, replayed with a residual correction that cuts tracking error from about 4 mm to 1.2 mm. Deployed on a Franka Research 3 with a wrist camera.

Results

Five simulated tasks (Isaac Lab Factory and ManiFeel, 150 trials each): 86.9% average success, versus 64.1% with naive end-to-end training, 58.7% with a frozen ViT, 74.2% for UWM and 68.7% for a visuo-tactile diffusion policy.
The tactile-exploration tasks gain most: 93.3% on BallSort and 85.3% on ObjectSearch, where the frozen-encoder variants reach 14.7-40.0%.
Real engine, 20 trials per task: bolt screwing succeeds 16/20 (UWM 10/20, frozen ViT 6/20) and nut mating 8/20 (UWM 5/20, frozen ViT 3/20).

Limitations

Force is processed at a low rate and may miss high-frequency contact dynamics, the per-modality loss weights are tuned by hand, and blocking each modality’s own denoising loss may make the latent space harder to diffuse. The comparisons with the MDF baselines use per-task maxima across seeds, so they do not fully isolate the objectives. Nut mating still fails in 12 of 20 real trials.

Videos

A bolt-screwing trial, with the live GelSight image in the corner.

A nut-mating trial on a fixed bolt of the engine.

Figures

From Figure 1: Prior world-action models predict future observations encoded by a frozen pretrained encoder; MDF++ le... — From Figure 1: Prior world-action models predict future observations encoded by a frozen pretrained encoder; MDF++ learns task-aware visuo-tactile representations end-to-end through cross-modality prediction.

From Figure 2: Training overview. The encoders and a latent diffusion world-action model are trained jointly on struc... — From Figure 2: Training overview. The encoders and a latent diffusion world-action model are trained jointly on structured noise patterns (policy, world-action model, forward and inverse dynamics, random masking), and each modality's denoising loss is blocked from updating its own encoder.

From Figure 3: Average stage-based scores on the real bolt-screwing and nut-mating tasks for MDF++, UWM and MDF++ wit... — From Figure 3: Average stage-based scores on the real bolt-screwing and nut-mating tasks for MDF++, UWM and MDF++ with a frozen ViT.

Abstract

Recently, world-action models (WAMs) have emerged as a powerful framework for robot manipulation. By jointly modeling action generation and future observation prediction, WAMs can act and anticipate the outcomes of the actions. However, existing WAMs typically rely on frozen pretrained encoders trained with reconstruction, which may not produce representations well suited for robot control. In this paper, we tackle the problem of learning physicallygrounded, control-relevant representations for WAMs. We propose MDF++, a visuo-tactile world-action model that learns latent representations of multisensory observations end-to-end with a diffusion backbone. While naive joint training can lead to feature collapse, MDF++ introduces a multi-task learning strategy that leverages cross-modal correlations to learn task-relevant representations. We evaluate MDF++ on a suite of contact-rich manipulation tasks, including realworld bolt screwing and nut mating, and show that it outperforms WAMs built on frozen pretrained representations by up to 50%. Additional videos are available on our website: jointmdf.github.io.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong