Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

MAPO: Mixture of Experts with Alternating Policy Optimization for Universal Humanoid Motion Tracking

Seonghyeon Lim, Hyeonwoo Lee, Mincheol Kim, Kimin Lee, Hyun Myung

November 2026

TL;DR MAPO trains a mixture-of-experts humanoid motion tracker by clipping the router and expert updates separately, since PPO's single KL measure misjudges how much each part changed. It reaches 86.6% tracking success versus 83.4% for PPO with the same MoE, and consistently tracks 182 motions where PPO with MoE fails (the reverse happens for 1).

What they do

Universal humanoid motion trackers often use a dense mixture-of-experts (MoE) policy, where a router mixes the outputs of several experts. The authors show that when such a policy is trained with PPO, the KL divergence of the mixed policy misjudges how much the router and the experts actually changed. Clipping then either blocks admissible updates or lets aggressive ones through. MAPO avoids this by regulating the router and expert updates separately.

How it works

  • Divergence misestimation. For a Gaussian MoE policy, the measured KL equals the router and expert terms plus a cross term whose sign cannot be controlled. Overestimation grows more common with more experts; underestimation dominates with fewer.
  • Alternating updates. For the expert update, the mixing weights are held at their rollout values; for the router update, the expert outputs are held. Each gets its own clipped surrogate loss, both computed from the same rollouts in one update step.
  • Teacher-student. A 6-expert MoE teacher with privileged observations is trained with MAPO in Isaac Lab on AMASS and LAFAN1 motions retargeted to a 29-DoF Unitree G1. DAgger then distills it into a monolithic student for the real robot.

Results

  • Training set: 86.6% success and 147.4 mm global body position error, against 83.4% and 153.1 mm for PPO with MoE and 81.8% for monolithic PPO. MAPO also beats PPO with MoE with 4, 8 and 16 experts.
  • Held-out OMOMO motions: 85.5% success, against 82.8% for PPO with MoE, 83.4% for HOVER and 57.9% for the released TWIST 2 checkpoint.
  • Across three seeds, MAPO consistently tracks 182, 155 and 191 motions that PPO with MoE consistently fails; the reverse happens for 1, 1 and 0. Success is 87.0% vs 82.9%.
  • The gain holds with a halved clip range for MAPO, a matched entropy coefficient and fixed learning rates, and against PPO variants with a KL penalty or early stopping.

Limitations

Expert specialization could be pushed further. The analysis assumes dense softmax routing with a fixed standard deviation per update, so it does not cover learned per-expert variances or top-k sparse routing. Holding one component fixed biases the router gradient. Table 1 comes from a single training seed, and the real G1 deployment is shown only qualitatively.

Figures

From Figure 1: MAPO tracks complex whole-body motions where PPO with MoE fails, such as (a) sitting in mid-air, (b) a...
From Figure 1: MAPO tracks complex whole-body motions where PPO with MoE fails, such as (a) sitting in mid-air, (b) abrupt full-body turns, (c) deep crouching and (d) one-footed spinning; red spheres are the reference targets. Surrounding frames show the distilled student on a real Unitree G1.
From Figure 3: Update mechanisms of MAPO and PPO, with the router in pink, experts in blue and the mixed policy in bl...
From Figure 3: Update mechanisms of MAPO and PPO, with the router in pink, experts in blue and the mixed policy in blue-green. (a, c) MAPO measures the KL divergence of each component separately. (b, d) PPO measures it for the mixed policy, so its clipping can block admissible updates or miss aggressive ones.
From Figure 4: Mean training reward of MAPO, MAPO with a fixed router, PPO with MoE and monolithic PPO.
From Figure 4: Mean training reward of MAPO, MAPO with a fixed router, PPO with MoE and monolithic PPO.
Abstract

Universal humanoid motion tracking is essential for enabling humanoid robots to perform diverse tasks in human environments. To capture the wide diversity of human motions, mixture-of-experts (MoE) policies have been widely adopted, leveraging dynamic routing to enhance policy expressiveness. However, we identify that divergence between old and updated policies can be misestimated during policy-gradient updates of MoE policies, due to their dynamic routing. Because most policy-gradient algorithms regulate policy updates based on the divergence, this misestimation can lead to either overly conservative or excessively large updates, which can yield a suboptimal policy. In this paper, we propose Mixture of Experts with Alternating Policy Optimization (MAPO), which alternately optimizes the router and the experts to avoid the divergence misestimation. MAPO consistently tracks 182 motions that PPO with MoE fails to track, whereas PPO with MoE tracks only one motion that MAPO fails to track. MAPO also attains the highest tracking success rate among the compared methods and the lowest tracking error across most error metrics, demonstrating its superior performance. Videos can be found on the project website: https://mapo-corl2026.github.io/MAPO/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.