Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

From Festival Video to the Stage: A Sim-to-Real Ladder for Long-Horizon Expressive Whole-Body Dance on a Commercial Humanoid

Hesham Shehata

November 2026

TL;DR A video-to-robot pipeline puts a 43-second Awa Odori festival dance on a Unitree G1, with each of eleven policy versions graded on hardware. Drift-aware observation noise and scripted edits of the reference motion take it from early falls to a continuous 9.7-minute rope-free performance, all trained on one RTX 5070.

What they do

The paper takes one ordinary monocular video of Awa Odori, a Japanese festival dance, and deploys the full routine on a 29-DoF Unitree G1 EDU, free-standing on a hard floor. It is written as a deployment study: eleven policy versions, each changing one variable relative to its parent and each graded on the real robot.

How it works

  • Pipeline. PromptHMR recovers SMPL-X motion from the video, GMR retargets it to the G1, and a per-motion BeyondMimic tracking policy is trained with PPO in Isaac Lab (2,048 environments), then exported to ONNX for the onboard RoboJuDo runtime. A fine-tune from a checkpoint takes 1.5-2 h on a single RTX 5070.
  • Artifact integrity. The training toolchain can silently corrupt the configuration that the exported artifact carries and the runtime applies to the robot; the authors add prevention, an automated validation and a repair step.
  • Drift-aware noise. A per-episode constant bias on the anchor pose and base velocity observations teaches the policy to stop chasing drifting odometry.
  • Reference editing. Small scripts on the motion CSV attenuate destabilizing joint groups while keeping the arms, append an ending that stops in the opening stance for handoff to the walking controller, and compose the clip into a long routine.

Results

  • The baseline (v1) fell within about ten seconds. With drift-aware noise, v3 ran 22 s rope-free on a hard floor; its parent v2 only completed on a foam mat with hand-steadying.
  • v5 completed the 43 s routine rope-free three times in a row, v6 the 46 s routine with the trained ending and automatic handoff, and v10/v11 a continuous 9.7-minute performance.
  • Live entry from the factory walking controller engaged 45-141 ms after the damp request, with 3.1-5.0 degrees of knee sag and no visible drop.

Limitations

Hardware evidence is single-digit trials per version, and floor, operator and session were not matched across versions. The robot ends the 9.7-minute routine about 1.5 m behind where it started, foam flooring makes every version fall, a second dance clip failed in capture under occlusion, and every motion edit needs its own fine-tune.

Figures

From Figure 2: Monocular capture on the public instructional video of the women's Awa Odori. The overlay shows the 2D...
From Figure 2: Monocular capture on the public instructional video of the women's Awa Odori. The overlay shows the 2D keypoint stage of the capture pipeline (person detection and keypoints) that precedes SMPL-X mesh recovery.
From Figure 3: Tracking training in Isaac Lab. The reference motion (body frames drawn as axes) is overlaid on the po...
From Figure 3: Tracking training in Isaac Lab. The reference motion (body frames drawn as axes) is overlaid on the policy's robot in one of the 2,048 parallel environments.
From Figure 5: The whole training campaign, thirteen runs over iterations 0 to 112,991. Top: mean reward, mean episod...
From Figure 5: The whole training campaign, thirteen runs over iterations 0 to 112,991. Top: mean reward, mean episode length (of 500), timeout fraction. Bottom: anchor position error, anchor orientation error, joint-velocity error. Only the drift-noise run (v3, 75k-85k) has a large anchor error, by design.
From Figure 12: Hardware deployment of the production policy (v10): the complete 9.7-minute routine performed without...
From Figure 12: Hardware deployment of the production policy (v10): the complete 9.7-minute routine performed without any protection.
Abstract

We present a pipeline for transferring Awa Odori, a traditional Japanese festival dance, from monocular video to a commercial Unitree G1 EDU humanoid. The pipeline combines human mesh recovery, kinematic retargeting, reinforcement learning for motion tracking in simulation, and onboard deployment via ONNX. Training ran on a single RTX 5070 GPU. An eleven-version deployment ladder evaluated on hardware examines three interventions. We detect and repair silent corruption of deployment artifacts, model state-estimator drift during training, and edit the reference motion to support stable execution and a controlled ending. The final policies achieved three consecutive rope-free completions of a 43-second routine, a 46-second performance with automatic handoff, and a continuous 9.7-minute rope-free performance composed from the captured motion. In a live handover, a single remote request let the robot start from and return to the manufacturer's walking controller on stage, with a measured torque-free window of 45-140 ms. We report the training dynamics and hardware outcomes and discuss failure boundaries involving compliant flooring and occlusion in monocular capture.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.