Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Online Adaptation to Latent Physical Properties for Learning from Demonstration

Kotaro Kajitsuka, Masaki Murooka, Fumio Kanehiro

November 2026

Real robot: lifting a weighted pouch and setting it level into the frame.

TL;DR A Diffusion Policy is conditioned on a latent physical-property estimate that is updated online from a learned dynamics model's prediction errors. This lets the robot regrasp or re-push objects that look identical but have different centers of mass. On a real xArm7 it raises pouch-placement success from 40.7% to 66.7% over a standard Diffusion Policy.

What they do

A robot cannot see where the weight sits inside a pouch or a box. It has to find out by interacting with the object. This paper learns that adaptation from teleoperated demonstrations in which the operator regrasps or moves the contact point after seeing how the object responds. Each object gets a learned latent “implicit dynamics parameter”, and a Diffusion Policy is conditioned on an online estimate of it.

How it works

  • Learning the latent. A VAE compresses camera images into 9-D features. An MLP dynamics model predicts the next 14 steps of these features (plus wrist force/torque for pushing) from recent observations, the planned actions and a per-object latent. The model and the latents are trained jointly on the demonstrations, without any measured physical parameters.
  • Policy. A Diffusion Policy is trained on the same demonstrations, each labeled with its object’s learned latent.
  • Online adaptation. At deployment the latent starts from the value of an object with plausible default properties, such as uniform density. Adam updates it on the prediction error every control step. The dynamics model and policy weights stay fixed.
  • Retraining. Autonomous rollouts, whether they succeed or fail, can be added to retrain the VAE and dynamics model. The policy is left untouched.

Results

  • Real xArm7, pouch placement with three center-of-mass positions: 66.7% success vs. 40.7% for a standard Diffusion Policy.
  • Box pushing with 11 weight positions (5 seen in demonstrations): 53.3% and 47.2% on seen and unseen positions, vs. 50.0% and 47.2% for Diffusion Policy. After retraining on rollouts this rises to 66.7% and 63.9%.
  • Simulated bar lifting: 58.6% overall vs. 49.0% for Diffusion Policy. A Diffusion Policy with a 16-step history drops to 40.5%.

Limitations

The latent is not always estimated reliably from brief contact. Slow or wrong estimates led to inserting a tilted pouch or to unnecessary contact changes. Adaptation to unseen center-of-mass locations stays weak, around 30% in simulation. Pouch placement was tested only on seen configurations, and weight varied along one dimension only, with a scalar latent.

Videos

Real robot: pushing a box whose hidden weight position is marked in red.
Simulated horizontal lifting of a bar with an off-center mass.

Figures

From Figure 1: Overview. (1) Teleoperated demonstrations include adaptive actions such as regrasping or changing cont...
From Figure 1: Overview. (1) Teleoperated demonstrations include adaptive actions such as regrasping or changing contact locations. (2) The same demonstrations train a dynamics model with object-specific physical representations, then a policy conditioned on them. (3) At deployment the robot starts from a reference representation, updates it from prediction errors and conditions its actions on the estimate.
From Figure 2: Training and deployment pipeline: dynamics-model training with a learned latent per object, Diffusion ...
From Figure 2: Training and deployment pipeline: dynamics-model training with a learned latent per object, Diffusion Policy training conditioned on that latent, and online updates of the latent during deployment.
From Figure 5: Real-robot success rates for Pouch Placement and Box Pushing.
From Figure 5: Real-robot success rates for Pouch Placement and Box Pushing. "Proposed + retraining" retrains the VAE and dynamics model on an augmented dataset while keeping the policy fixed. Error bars are approximate 95% Wilson confidence intervals pooled over three training seeds.
Abstract

Manipulating visually identical objects with different physical properties requires adapting actions based on information acquired through interaction. We address the problem of learning from demonstrations how to adapt manipulation actions, such as regrasping and changing contact locations, based on physical-property information acquired during interaction. To this end, we propose a learning-from-demonstration method that learns latent representations of physical properties needed to predict object responses and conditions a policy on online estimates of these representations. A dynamics model and object-specific latent representations are jointly learned from demonstrations that include behaviors adapting to the objects' physical properties, and the learned representations are then used to condition an imitation policy. During execution, the dynamics model and policy weights remain fixed, while the latent representation is updated based on the discrepancy between predicted and actual observations, allowing interaction with the object to inform action generation. Real-world and simulation experiments with objects whose centers of mass differ suggest that online adaptation through latent representations of physical properties can improve task success rates. However, in simulation, success rates on center-of-mass locations absent from the demonstrations remained around 30%, indicating limited adaptation to unseen physical properties. Project page: https://isri-aist.github.io/latent-dynamics-project-page/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.