HLO-AMP: Hierarchical Loco-Manipulation from Object-Aware Motion Priors

Jean-Pierre Sleiman, Bilal Hammoud, He Li, Jonathan Foster, Sylvain Bertrand, Jessica K. Hodgins, Farbod Farshidian

November 2026

TL;DR A Unitree G1 learns bimanual box pick-and-place from a low-level controller trained to imitate human motion capture with the box in the simulation, steered by a high-level RL policy. One deployed policy carries boxes up to 7 kg and recovers from drops. In simulation it succeeds 80.7% of the time, vs 50.3% for a whole-body controller trained without the object.

What they do

Humanoid loco-manipulation stacks usually put a task policy on top of a generic whole-body controller. That controller is trained without the object, with random end-effector forces standing in for the payload. HLO-AMP instead trains the low-level controller to imitate retargeted human motion capture of box handling, with the box physically in the simulation. A high-level RL policy then steers it through a full bimanual pick-and-place on a Unitree G1.

How it works

Motion library. One person’s MoCap of carrying, picking up and placing a box, retargeted to the G1 and re-solved for 10 box variants (30 clips), plus a walking clip.
Low-level prior. A PPO motion-imitation policy tracks the robot and box references but does not observe the box. It takes a compact 12-D command: base height, roll/pitch, planar velocity and both hand positions. Noise on this command during training lets the high-level policy steer it later.
High-level policy. It is reference-free and observes proprioception, the six box face centers and the goal. Its commands are clipped to the range seen in the low-level training data. A five-phase reward scheduler (walk, lift, carry, place, idle) is used only during training.
Deployment. Zero-shot sim-to-real, with the box pose from external motion capture.

Results

Hardware: every condition succeeded in 3/3 trials, including recovery after the box was pushed out of the grasp, a 60 cm cube larger than any training box and multi-box sequences. The exceptions were 7 kg (2/3) and 8 kg (0/3, thermal shutdown or a preemptive stop).
Simulation, 20 s episodes: 80.7% success vs 50.3% for a whole-body controller trained without the object and 77.1% (with a 78.0% drop rate vs 26.8%) for one trained with a force surrogate. An end-to-end policy reaches 3.6%.
Removing any one component (observation noise, the box in low-level training, targeted resets or command clipping) drops success to 3-16%.

Limitations

Box pose comes from motion capture, so on-board perception is untested. Behaviors are limited to the four primitive families. Each new task needs new demonstrations, a new phase graph and new rewards. Naturalness is measured only by a proxy index, not a user study.

Videos

Figures

From Figure 1: Retargeted MoCap builds a motion library of locomotion, walking-with-box, pickup and placement. A low-... — From Figure 1: Retargeted MoCap builds a motion library of locomotion, walking-with-box, pickup and placement. A low-level motion-imitation policy is trained with the box in the scene, conditioned on a noisy task-space command. A reference-free high-level policy then drives the frozen low-level policy from proprioception, box face centers and a sparse goal, and the result deploys zero-shot on hardware.

From Figure 2: Per-joint torque during the 7 kg run, normalized by each joint's effort limit, with snapshots of the l... — From Figure 2: Per-joint torque during the 7 kg run, normalized by each joint's effort limit, with snapshots of the lift (about 9 s) and the carry (about 11 s).

Abstract

Hierarchical pipelines for humanoid loco-manipulation typically pair a task-agnostic whole-body controller (WBC) with a downstream task policy. Such WBCs are often trained through manual reward shaping, which yields visibly machine-like motions; more consequentially, they are trained without the manipulated object in the scene. Object dynamics are instead approximated through randomized end-effector force disturbances, a proxy that fails to reproduce the distributed multi-contact wrench a real payload applies across the arms and torso, with the gap becoming critical near the platform's payload limit. In this work, we train the low-level (LL) controller itself as an object-aware motion prior: a single policy trained via motion imitation on retargeted motion-capture (MoCap) references that span the core motion primitives of the downstream task, with the manipulated object physically present in the scene. This single stage inherits natural human-like coordination from the references and absorbs the realistic contact wrench at its source, while exposing the same compact task-space command interface used by command-tracking WBCs. A reference-free high-level (HL) policy is then trained on top, with its action channel anchored to the empirical envelope of the low-level training data so that it generalizes the priors to a long-horizon multi-stage task while structurally preventing queries that would push the low level off-distribution. We instantiate the framework on a Unitree G1 for bimanual box pick-and-place: the same deployed policy generalizes across object masses and dimensions, initial and goal poses, and reactively recovers from intentional drops mid-carry. Video: https://youtu.be/viR_ULnRDQQ.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong