STEP: Spatiotemporal Training with Explicit Prediction for Perceptive Humanoid Locomotion

Youhao Lyu, Chenghao Sun, Zhenxuan Ding, Shuting Zhang, Ziyang Dong, Wenhao Cui

November 2026

TL;DR STEP trains a depth-based humanoid locomotion policy with two training-only losses, cross-modal masked prediction and action-conditioned next-state prediction, that shape its fused body-terrain state. On a 23-DoF Noetix E1 with onboard sensing it completes 98 of 100 hardware trials over stairs, a 40 cm platform, a 70 cm gap and stepping stones.

What they do

STEP trains a depth-based humanoid locomotion policy in a single stage: the PPO policy and its multimodal state representation learn together. Two training-only prediction losses tell the encoder which body-terrain relationships to keep. The policy runs on a 23-DoF Noetix E1 humanoid with an onboard depth camera.

How it works

Multi-query fusion. A five-frame proprioceptive history (5 tokens) and a 16x24 depth image (96 tokens from a CNN) form one shared context. Four learned queries read it through two cross-attention blocks and are pooled into a 64-dimensional policy state.
Cross-modal prediction. With depth masked, the state must predict an EMA teacher’s depth features; with proprioception masked, its proprioceptive features.
Next-state prediction. A small predictor maps the current state and the executed action to the teacher’s next fused state. SIGReg prevents latent collapse, and prediction heads and teachers are dropped at deployment.
Training runs in Isaac Lab with PPO, a privileged critic, an AMP motion prior, a separate velocity estimator and a Soft-MoE actor.

Results

Simulation (gaps, stairs up and down, stepping stones at two difficulty levels): STEP has an overall mean of 95.17% across success and traversal rates. That compares with 91.02% without the prediction losses, 69.98% with one query and 86.17% for a proprioception-queried cross-attention baseline. On the hardest stepping stones, success is 74.9% versus 57.2% without prediction and 43.5% for the baseline.
With a 20 ms depth delay, STEP reaches 88.6% on stepping stones; removing the next-state loss drops this to 54.1%, removing cross-modal prediction to 0%.
Hardware: 98 of 100 trials. It goes 20/20 on 20 cm stairs, the 40 cm platform, stepping stones and backward 15 cm stairs without a rearward view, and 18/20 on the 70 cm gap.

Limitations

If depth stops updating altogether, every variant, STEP included, fails on gaps and stepping stones. Prediction looks only one step ahead, and the policy keeps no memory of terrain it has already seen. Hardware tests use 20 trials per task, and both failures were on the gap.

Figures

From Figure 1: STEP on the real humanoid, climbing 20 cm consecutive stairs, stepping onto a 40 cm high platform and ... — From Figure 1: STEP on the real humanoid, climbing 20 cm consecutive stairs, stepping onto a 40 cm high platform and crossing a 70 cm gap.

From Figure 2: Four learned queries aggregate proprioceptive and depth tokens into a 64-dimensional policy state, whi... — From Figure 2: Four learned queries aggregate proprioceptive and depth tokens into a 64-dimensional policy state, which a Soft-MoE actor uses together with a separately estimated velocity. The lower row shows the training-only objectives: action-conditioned prediction of the next EMA state and bidirectional masked cross-modal prediction. They are dropped at deployment.

From Figure 4: Real-world traversal sequences on a 70 cm gap, 20 cm stairs, a 40 cm platform, stepping stones and bac... — From Figure 4: Real-world traversal sequences on a 70 cm gap, 20 cm stairs, a 40 cm platform, stepping stones and backward stairs with no rearward view. On the backward stairs a first foot collision is followed by collision-free steps.

Abstract

For perceptive humanoid locomotion, the relevance of terrain geometry depends on the robot's body state and how both evolve under its actions. Existing work has advanced terrain-aware sensory fusion and cross-modal prediction, improving how visual and proprioceptive information is combined and related. However, feature aggregation alone does not specify which cross-modal relationships should be learned, while prediction between simultaneous observations does not explicitly constrain their action-dependent evolution. We present Spatiotemporal Training with Explicit Prediction (STEP), a one-stage framework that jointly supervises cross-modal relationships and action-dependent state evolution within a shared control representation. Multiple learned queries aggregate depth and proprioceptive tokens into a compact policy state. Bidirectional masked prediction uses each visible modality to predict latent features of the omitted modality, explicitly supervising terrain-body relationships. An action-conditioned predictor maps the current policy state and executed action to the next fused state, with an exponential-moving-average target defined directly in the policy representation space. Simulation comparisons show overall improvements in terrain traversal over evaluated fusion alternatives and ablations, while depth-dropout and delay experiments demonstrate the benefits of both predictive objectives under degraded visual input. Deployed on a 23-DoF Noetix E1 humanoid with onboard sensing, STEP succeeds in 98 of 100 hardware trials across stairs, a 40 cm platform, a 70 cm gap, stepping stones, and backward stair traversal without rearward terrain visibility.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong