Coordinating Humanoid Loco-Manipulation through Action-Conditioned Latent Foresight

Ju Dong, Xinyu Xiao, Yiming Zhou, Tianshuo Yang, Xuerong Liu, Lei Zhang, Jian Chen, Haocheng Zhao, Yu Fu, Yujie Wei, Diwen Zheng, Alois Christian Knoll, Angela P. Schoellig, Ping Luo, Jianwei Zhang

November 2026

Real Unitree G1 on Dispose of Trash, picking an item off the table and turning to the pedal bin.

TL;DR A humanoid policy samples several whole-body and hand command sequences, predicts in latent space where each would lead, and executes the one that makes the most visual progress and is most temporally consistent. On a Unitree G1 it succeeds 85.7% on seven tasks (65.7% with unseen objects) versus 70.0% (48.6%) for GR00T N1.7.

What they do

A plausible-looking humanoid command sequence can still drop a held object or leave a task unfinished. This paper samples several body-motion and hand command sequences, predicts in latent visual space where each leads, and runs the first commands of the best one before replanning. Tested on ten SIMPLE simulation tasks and seven real Unitree G1 tasks.

How it works

Proposals. Qwen3-VL-2B encodes the egocentric image and instruction; an Action DiT with flow matching samples 8 command chunks. On the real robot, each command is a 64-dimensional SONIC motion latent plus 14 hand joint targets.
Visual progress. A V-JEPA 2.1 encoder, fine-tuned on 500+ hours of G1 teleoperation in homes, embeds images. An endpoint predictor estimates the features reached after the first 20 commands and scores the predicted reduction in distance to the current subgoal, one of 5 subgoal images extracted from demonstrations.
Temporal consistency. A GRU predicts each next motion and hand command from the state and previous commands; its prediction error penalises inconsistent sequences.
Action consistency regularization. During training, a decoder must recover the first demonstrated command from both observed and predicted feature changes, so the predictor keeps action-relevant information.

Results

SIMPLE (ten tasks, 30 trials each): 89.7% success, versus 82.0% for DiT4DiT and 73.7% for Psi-0.
Real G1 (seven tasks, 10 trials per condition): 85.7% standard and 65.7% with unseen objects, versus 70.0% and 48.6% for GR00T N1.7. Load Washer accounts for most of the gap: 10/10 versus 2/10.
Ablations on four SIMPLE tasks: action consistency regularization raises success from 55.0% to 65.8%; the joint temporal prior reaches 80.8%, versus 65.8% for visual-only scoring and 70.0% for a second-order smoothness penalty.
Onboard the G1’s Jetson Orin NX, inference runs at 0.73 Hz using 12.84 GB of memory.

Limitations

Failures include partly opened articulated mechanisms (faucet, bin lid) and losing grip on a soft toy. Scoring depends on task-specific subgoal images from demonstrations, each task trains its own policy, and onboard inference is slow at 0.73 Hz.

Figures

From Figure 1: Place Basket on Table with GR00T N1.7 (the basket falls) and with this method (the basket ends on the ... — From Figure 1: Place Basket on Table with GR00T N1.7 (the basket falls) and with this method (the basket ends on the table). Each sampled command sequence is scored by predicted visual progress toward a subgoal and by temporal-consistency energy; the robot executes the start of the best one and replans.

From Figure 2: Architecture. A Qwen3-VL and Action DiT generator proposes sequences; a V-JEPA 2.1 endpoint predictor ... — From Figure 2: Architecture. A Qwen3-VL and Action DiT generator proposes sequences; a V-JEPA 2.1 endpoint predictor scores visual progress toward the active subgoal and a joint temporal prior scores consistency. In training, an inverse action decoder reconstructs the first demonstrated command from observed and predicted feature changes.

From Figure 4: Water Plant and Load Washer under standard and unseen object configurations; with a reversed spout or ... — From Figure 4: Water Plant and Load Washer under standard and unseen object configurations; with a reversed spout or the T-shirt on the left, the robot switches to its left hand.

Abstract

Humanoid loco-manipulation requires coordinated body motion and hand interaction over time, and a plausible-looking command sequence can still drop a held object or leave an interaction incomplete. We present action-conditioned latent foresight, a latent world-action model (WAM) that evaluates proposed command sequences before execution. An action generator proposes joint motion-hand command sequences from vision, language, and the robot state. An action-conditioned endpoint predictor estimates the visual features each sequence prefix reaches and measures progress toward a demonstration-derived subgoal, while a joint temporal prior scores temporal consistency given the robot state and preceding commands. The robot executes the first commands of the best-scoring sequence and replans from the next observation. During training, action consistency regularization reconstructs the first demonstrated command from observed and predicted feature changes, encouraging the predictor to retain action-relevant information. On ten tasks in the humanoid loco-manipulation simulation benchmark SIMPLE, our method achieves 89.7% average success, compared with 73.7% for Psi-Zero and 82.0% for DiT4DiT. On seven real-robot tasks, it achieves 85.7% under standard conditions and 65.7% under unseen object configurations, while GR00T N1.7 achieves 70.0% and 48.6%, respectively. Project page: https://ju6276.github.io/Action-Grounded-Foresight/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong