HITCH: Teaching a Vision-Language Model to Operate a Humanoid through a Whole-Body Harness

Qingzhou Lu, Chenxin Liu, Xinming Wang, Liu Cao, Yanjiang Guo, Mengdi Xu, Jianyu Chen

November 2026

TL;DR A harness lets a VLM measure 3D points from depth and command a Unitree G1's pelvis, torso, hands, walking and gaze, and Qwen3.8-27B is fine-tuned on teacher trajectories from five base tasks. It solves 51% of unseen SIMPLE episodes (95% with five trajectories per task) and carries a box on a real G1 in 13 of 20 tries with no real training data.

What they do

Pretrained VLMs understand 3D space but know little about how a humanoid body moves, and existing humanoid agents mostly call skill libraries that leave leg and torso posture to the controller. HITCH gives a VLM two interfaces to a Unitree G1, one to measure the scene and one to command each body part. It then fine-tunes Qwen3.8-27B on trajectories of teacher policies that solve five base tasks through the same interfaces, so the model learns where to stand, how low to crouch and what to measure before acting.

How it works

Perception. The agent picks points in the image from a RealSense D455 on an added 2-DoF head mount. Depth, refined with LingBot-Depth, returns their 3D positions in a robot-centred frame.
Execution. One command sets pelvis height, torso pitch and yaw, palm waypoints, fingers, walking and gaze. A planner solves IK into a whole-body reference that a HEFT tracker follows at 50 Hz. The tracker is further trained on the planner’s references.
Teachers and training. Code teachers (written with Claude Opus 5.5) act only on what the agent can observe. 500 successful trajectories per base task (pick-and-place, carrying, navigation, cabinets, room doors) fine-tune the language model, with the vision encoder frozen.

Results

With the further-trained tracker, teachers succeed in 97.9% of episodes, against 27.9% with SONIC and 12.9% with AMO.
Base-task training alone raises success from 3.3% to 51.1% on SIMPLE and from 16.3% to 42.5% on held-out HumanoidArena. Adding 5 SIMPLE trajectories per task reaches 95.0%. Qwen3.8-Max without fine-tuning gets 12.2% on SIMPLE.
Without the perception interface, base-task SIMPLE success falls to 2.8%.
Real G1 with rubber hands and no real training data: 13/20 box carries and 6/10 navigation episodes.

Limitations

The planner does not support motions such as sitting or stepping over obstacles. Each command runs to completion before the agent observes again, so it cannot react mid-motion. Adding SIMPLE trajectories lowers HumanoidArena success, and after fine-tuning, reasoning lowers SIMPLE success because the agent builds on wrong point readings. Hands without finger control limited the real tests to two tasks.

Figures

From Figure 1: A fine-tuned VLM controls a G1 humanoid through the harness. In a SIMPLE episode it measures the box a... — From Figure 1: A fine-tuned VLM controls a G1 humanoid through the harness. In a SIMPLE episode it measures the box and the basket by choosing points in its head-camera images, then lifts the box, turns, walks to the other table and lowers its hand into the basket. Right: other tasks in SIMPLE, HumanoidArena and on a real G1, and SIMPLE success of the pretrained model, the base-task model and the base-task model with 5 trajectories per SIMPLE task.

From Figure 2: Overview. In each turn the agent (Qwen3.8-27B) makes one call. A perception call returns the 3D positi... — From Figure 2: Overview. In each turn the agent (Qwen3.8-27B) makes one call. A perception call returns the 3D positions of points it selects in the image, measured from depth; an execution call sets the body, hands, fingers, walking and gaze, which a planner turns into a whole-body reference for the HEFT tracker. Teacher code policies act through the same harness on five base tasks and six SIMPLE tasks, and the agent is fine-tuned on their multi-turn trajectories.

From Figure 5: Episodes in time order, with the call that returned each frame below it. Left: teacher episodes of thr... — From Figure 5: Episodes in time order, with the call that returned each frame below it. Left: teacher episodes of three base tasks. Right: the agent on SIMPLE, on HumanoidArena, and carrying a box on the real G1 after training on the base tasks only.

Abstract

Enabling vision-language models (VLMs) to control humanoid robots requires both reliable whole-body interfaces and knowledge of how to use the body that carries over across tasks. We present HITCH (Harness Interfaces To Command Humanoids), a harness in which perception returns the 3D position of any point the VLM selects in its image, and execution turns commands for the pelvis, torso, hands, walking, and gaze into whole-body motion with a learned tracker. We find that both measuring the scene and executing commands accurately raise task success. We then fine-tune a VLM on trajectories in which teacher policies solve five base tasks through HITCH, so that it learns how to operate the humanoid. Without task-specific trajectories, the fine-tuned model completes 51% of SIMPLE episodes and 42.5% of HumanoidArena episodes, and adding five trajectories per SIMPLE task raises its SIMPLE success to 95%. On a real G1, it carries a box and walks to a goal without real training data.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong