HINT-VLM: Humanoid Interactive Navigation over Rough Terrain with Embodiment-Grounded Vision-Language Model

Ziwon Yoon, Seok Hwan Kim, Yiyu Chen, Quan Nguyen, Lu Gan, Ye Zhao

November 2026

Real Unitree G1 pushes a cart out of its way, shown next to the planner's map and chosen path.

TL;DR A Unitree G1 humanoid decides before contact whether to push an obstacle aside or walk around it, using a VLM's movability guess corrected by a Gaussian process fit to simulated pushes. It reaches the goal in 95.5-100% of simulated rough-terrain trials and cuts real-world navigation distance from 16.8 m to 10.3 m.

What they do

Navigation Among Movable Obstacles (NAMO) has mostly been studied on flat floors, with robots that learn whether an object moves only by pushing it. HINT-VLM lets a humanoid on rough terrain estimate from a camera image the probability that it can push an object aside, then plan with that probability: push through, or detour.

How it works

VLM-GP movability. The Yes/No logit difference of a pretrained Qwen3.5-9B VLM gives a semantic prior. A residual Gaussian process over mass, friction, size and material corrects it, fit to about 20,000 simulated pushes of roughly 400 objects in IsaacSim with the deployment controller. Far from the training data the estimate reverts to the VLM prior.
Terrain under the object. The Grounded-SAM 2 object mask is cut from the depth image and inpainted with LaMa, so the ground beneath the object can be checked for traversability.
Expected-cost planning. Each interaction route is scored as the cost after a successful push and the fallback cost after a failed one, weighted by the movability probability, and compared with plain detours.
Execution. A finite state machine switches from path tracking to a forward-arm pushing behavior targeting 0.5 m/s.

Results

Movability prediction on 83 real objects: AUROC 0.868 for VLM-GP, against 0.827 for the raw VLM, 0.817 for LoRA fine-tuning and 0.736 for a DINOv3 predictor.
Simulation, 44 trials per environment: navigation success of 95.5%, 97.7% and 100% in Garden, Construction and Park. The non-interactive STATE-NAV finds no path in Garden, and the end-to-end VAMOS reaches 52.3%, 63.6% and 11.4%.
Skipping terrain reconstruction drops Park success to 86.4%.
Real G1, eight indoor trials: all three tested planners reach the goal, but HINT-VLM takes 23.2 s and 10.3 m, against 27.6 s and 16.8 m for STATE-NAV.

Limitations

Interaction is a simple forward push that does not optimize where the object ends up. Scaling to dense clutter is open, since object instances must be told apart and each one queried through the VLM. Real-world tests are few (two trials per object) and the authors call them preliminary.

Videos

Simulated construction site, where the robot pushes a wooden chair aside to take the shorter route.

Figures

From Figure 1: Overview of HINT-VLM. The proposed framework uses an embodiment-grounded VLM-GP to predict object mova... — From Figure 1: Overview of HINT-VLM. The proposed framework uses an embodiment-grounded VLM-GP to predict object movability from visual observations, maps objects and reconstructs the underlying terrain, and incorporates the resulting information into an uncertainty-aware global planner. A finite-state machine switches among path tracking, interaction, and failure recovery.

From Figure 2: Overview of the proposed VLM-GP training and inference. The residual GP is trained on simulated robot ... — From Figure 2: Overview of the proposed VLM-GP training and inference. The residual GP is trained on simulated robot interaction outcomes and ground-truth physical attributes, while perceived attributes are used at inference for embodiment-grounded movability estimation.

From Figure 4: Illustration of uncertainty-aware path planning. The solid green path leads to the object, while the d... — From Figure 4: Illustration of uncertainty-aware path planning. The solid green path leads to the object, while the dotted and dashed green paths represent the routes following successful and failed interactions, respectively. The orange path denotes a non-interactive detour.

From Figure 7: Outdoor testing environment (22 m x 41 m) with a shortcut path blocked by a chair. The green path indi... — From Figure 7: Outdoor testing environment (22 m x 41 m) with a shortcut path blocked by a chair. The green path indicates the route selected by the robot, which clears the chair to take the shortcut.

Abstract

Humanoid robots have achieved significant progress in locomotion and manipulation, enabling traversal over rough terrain and richer interaction with the environment, such as moving obstacles. However, most traditional navigation methods assume non-interactive environments and focus only on traversal. While Navigation Among Movable Obstacles (NAMO) approaches relax this assumption, prior efforts have largely focused on flat environments and reactive interaction, rather than predicting object movability and planning ahead. These limitations become critical in outdoor rough-terrain scenarios, where robots must jointly reason about terrain traversability and object movability. In this work, we address NAMO for humanoid robots in cluttered outdoor rough terrain by proposing a hybrid planning framework that combines semantic priors from a Vision-Language Model (VLM) with embodied robot interaction data through a Gaussian process (GP) residual model. The GP corrects the VLM-derived movability prior to produce embodiment-grounded probabilistic movability estimates, which are then incorporated into hierarchical uncertainty-aware planning. The proposed approach allows humanoids to jointly leverage locomotion and manipulation for efficient and safe navigation, representing the first solution to NAMO in rough terrain through predictive movability reasoning and probabilistic VLM integration.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong