← Paper Digest

PhysVAM: Physics-oriented Future Value Action Model for Dynamic Object Manipulation

Yuxiang Yang, Jingbo Zhang, Jiaru Zhong, Junliang Chen, Yirui Zhou, Guangming Wang, Haitao Lin, Huaiyuan Xu, Yi Wang, Lap-Pui Chau

November 2026

Real-world turntable tasks at original speed, placing moving bread on a plate and a moving cup into a box.

TL;DR PhysVAM grasps moving objects with a policy that predicts how task progress will change 4 to 32 steps ahead and uses those predictions to steer its action generator. It averages 58.8% success on ten simulated moving-object scenes and 64.2% on a real arm with conveyor, turntable and rolling objects, 27.0 and 27.5 points above the best baseline (PUMA).

What they do

When objects keep moving while the robot perceives and acts, a grasp that looks good now can still miss. Instead of predicting future images, PhysVAM predicts future task progress: how much an upcoming interaction will advance or set back the task. Those predictions condition the action generator.

How it works

  • Inputs. Language, proprioception and three camera views, plus four steps of 3D object position with velocities from finite differences, and the robot’s recent actions.
  • Two coupled DiTs. A 1B-parameter ActionDiT, initialized from the Wan2.2 video DiT as in Fast-WAM, outputs 32-step action chunks. A smaller ValueDiT has one learned query per horizon (4, 8, 16, 32 steps) and predicts the task value at each.
  • Value targets. Each episode gets a value curve in [-1, 1] whose sign is the final outcome. Failed trajectories train only the value branch, successes train both.
  • Guidance. Value-query features are injected into three ActionDiT layers through gated cross-attention.

Results

  • Simulation (ten object scenes, objects moving at 0.2-0.4 m/s or, in five scenes, at 0.4 m/s and faster; 20 trials per scene over three seeds): 58.83% average success and 8.23 s completion time, against 31.83% for PUMA and 17.33% for DynamicVLA. Adding the value branch to PUMA raises it to 41.50%.
  • Ablations: no value branch 50.50%, scalar value conditioning 40.83%, current-value instead of future-value targets 46.50%, no object-motion history 17.17%.
  • Real AgileX PiPER arm, 20 trials per object: 72.5% on the conveyor belt, 65.0% on the turntable and 55.0% on rolling objects, 64.17% overall against 36.67% for PUMA.

Limitations

The policy learns only from offline trajectories, and on the real robot object motion comes from FoundationPose tracking, which needs a generated object mesh and color-based segmentation. Real-world tests cover five objects on one arm. The authors list online improvement, more kinds of object motion, multi-object interactions and longer tasks as future work.

Videos

The real arm picks a cube off a moving conveyor belt and drops it in a basket.
A simulated can-to-plate rollout next to the predicted future values at four horizons.

Figures

From Figure 1: (A) Visual future prediction versus value-based future modeling: PhysVAM predicts task values at sever...
From Figure 1: (A) Visual future prediction versus value-based future modeling: PhysVAM predicts task values at several future horizons to guide action generation. (B) The predicted values drop after a missed grasp and rise during recovery and a successful lift. (C) Success rates of the compared methods in simulation and on the real robot.
From Figure 2: Object and robot motion histories give temporal context, ValueDiT learns future task-value features at...
From Figure 2: Object and robot motion histories give temporal context, ValueDiT learns future task-value features at several horizons, and those features steer ActionDiT through gated cross-attention.
From Figure 6: Real-world success rates of DynamicVLA, PUMA and PhysVAM on conveyor-belt, turntable and rolling scene...
From Figure 6: Real-world success rates of DynamicVLA, PUMA and PhysVAM on conveyor-belt, turntable and rolling scenes, with example executions.
Abstract

Predicting future scene or object states does not necessarily reveal how robot-object interactions will advance or hinder a task. This challenge is particularly pronounced in dynamic robotic manipulation, where objects continue moving during perception and execution. Effective action generation therefore requires not only modeling object dynamics but also anticipating how future interactions will shape task progress. To this end, we propose PhysVAM, a Physics-oriented Future Value Action Model that represents anticipated interactions through their effects on future task progress. This value-based formulation provides task-directed foresight without reconstructing future visual states. Conditioned on current observations and historical object and robot motion, PhysVAM jointly learns action generation and multi-horizon future-value prediction. The value branch learns horizon-specific latent representations through future task-value prediction from successful and failed trajectories. These future-value representations are integrated into the action policy through gated cross-attention, guiding action generation toward interactions that facilitate task success. Extensive simulation and real-world experiments demonstrate superior task success, with absolute success-rate improvements of 27.0 and 27.5 percentage points over the strongest baselines in simulation and real-world tasks, respectively. Project page: https://metarobo.github.io/physvam.github.io/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.