ForceLDM: A Force-aware Latent Dynamics Model for Contact-Rich Manipulation

Peiyuan Tang, Yao Li, Xiaodong Zhang, Wuyang Zhang, Qin Xia, Yanyong Zhang, Zijiang Yang

November 2026

xArm 7 inserts a USB plug into a power strip.

TL;DR ForceLDM distills future force/torque and optical-flow features from a privileged teacher into a π0-based student policy that sees only current observations. On five real-robot contact-rich tasks it averages 87.1% success, 14.9 points above the strongest baseline.

What they do

Force-aware VLAs usually feed force/torque in as one more input and learn a direct observation-to-action mapping, while video world models predict future frames and mix control-relevant dynamics with appearance details. ForceLDM instead teaches a π0 policy to anticipate upcoming contact and motion in latent space. A teacher sees privileged future force and optical flow; a student that sees only current observations learns to match the teacher’s internal representations, and only the student is deployed.

How it works

Future dynamics encoding. An MLP turns the next 32 steps of 6D force/torque into one force token. RAFT optical flow between the current and future frame is colorized, compressed with SDXL-VAE and pooled by 16 learnable queries into motion tokens.
Teacher-student distillation. Two action experts share π0’s PaliGemma backbone and get the robot state and a 10-step force history. The student’s 17 learnable queries are aligned with the teacher’s force and motion representations at layers 8, 12 and 16 (cosine loss, stop-gradient on the teacher); both experts train with flow matching.
Progressive noise injection. Gaussian noise on the student’s queries ramps up between 30% and 70% of training to curb overfitting.

Results

xArm 7 with a wrist force/torque sensor, 80-100 SpaceMouse demonstrations and 20-30 trials per task.

In distribution: 87.1% average success vs 72.2% for Seer with force, 67.7% for TA-VLA and 41.1% for π0 without force; 85% on VGA plugging, where the best baseline reaches 50%.
Out of distribution (occlusion, unseen objects, color, height, material, size; 15 trials each): 68.9% vs 48.9% for the best baseline.
Ablations: force plus flow reaches 90.7%, force alone 68.0%, flow alone 70.7%. Removing the distillation loss drops 85.1% to 42.8%; removing noise injection drops it to 74.1%.
On a Piper-X with GR00T N1.5 and motor-current torque estimates: 70% vs 62.5% for TA-VLA and 20% for GR00T N1.5.

Limitations

Datasets are small (3.08 hours on the xArm), so whether the gains hold at scale is open. Optical-flow errors from occlusion or motion blur can corrupt the learned dynamics, and 2D flow carries no depth. ForceLDM trails TA-VLA on gear assembly (83.3% vs 93.3%) because of grasp failures near the workspace boundary.

Videos

USB insertion while a person moves the power strip by hand.

Gear assembly with an unseen gear color.

Figures

From Figure 1: (a) Force-aware VLAs take force feedback but do not model future interaction dynamics. (b) Video-based... — From Figure 1: (a) Force-aware VLAs take force feedback but do not model future interaction dynamics. (b) Video-based world action models predict the future visually, but often lack force or tactile feedback and may fit task-irrelevant visual details. (c) ForceLDM distills compact future motion and force cues from a privileged teacher.

From Figure 2: (a) Teacher and student both generate actions from current observations and the instruction; the teach... — From Figure 2: (a) Teacher and student both generate actions from current observations and the instruction; the teacher also receives future force and motion tokens, and its representations are distilled into the student's future queries. (b) The future dynamics encoder turns future force/torque and optical flow into compact force and motion tokens. (c) Progressive noise injection perturbs the student's future queries with scheduled noise.

From Figure 3: The five real-world tasks: USB insertion, gear assembly, box flipping, board wiping and VGA plugging.

From Figure 5: Probes on the frozen policy's hidden states predict future motion (orange-red arrows) at five stages o... — From Figure 5: Probes on the frozen policy's hidden states predict future motion (orange-red arrows) at five stages of USB insertion: approach, stuck, realign, insert and return. Bottom: measured contact force (blue solid) and predicted force (red dashed).

Abstract

Contact-rich manipulation requires fine-grained control and complex physical interactions with the environment, posing significant challenges for robot policy learning. Vision-only policies struggle with such tasks, as contact states and interaction forces are difficult to infer from visual observations alone. Existing methods mitigate this issue by incorporating force or tactile sensing. However, they often treat these signals as auxiliary observations without fully exploiting their role in modeling future interaction dynamics. To address this, we propose ForceLDM, a force-aware latent dynamics model that learns future contact and motion dynamics to guide action generation. Rather than directly predicting future observations, which can entangle task-relevant dynamics with irrelevant sensory details, ForceLDM learns a compact representation of future dynamics through knowledge distillation. Specifically, a teacher network with privileged future force/torque and optical flow signals extracts future dynamic features, which are then distilled into a student network trained only on current observations. This enables the student model to reason about upcoming contacts and physical dynamics without relying on future sensory inputs at deployment. To improve robustness and generalization, we further introduce a curriculum-based progressive noise injection strategy to mitigate over-reliance on future features. Experiments on five real-world contact-rich manipulation tasks demonstrate that ForceLDM outperforms state-of-the-art baselines, generalizes effectively to novel objects, and remains robust to environmental perturbations. Project page and code are available at https://tpy001.github.io/ForceLDM/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong