MoViC: Real-Time Humanoid Whole-Body Control from Monocular Video Streams
TL;DR MoViC drives a full-size humanoid from a single webcam: the operator is reconstructed at about 10 Hz, a learned extrapolator fills in 50 Hz references, and a transformer tracker conditioned on a compact frequency-domain motion code follows them. The tracker reaches 99% success versus 95% with a larger ten-frame preview, and the real robot carries a box and a bag under webcam teleoperation with no wearables.
What they do
Humanoid whole-body teleoperation usually needs motion capture or a VR headset. MoViC needs only a fixed RGB webcam. It reconstructs the operator’s body per frame, retargets the motion to the 24-DoF EngineAI PM01 and feeds it to a universal motion tracker trained with PPO on retargeted human motion datasets. The same policy tracks dance and teleoperates object manipulation with no task-specific training.
How it works
- Video to reference. YOLO11-pose finds the person, SAM 3D Body reconstructs the mesh at about 10 Hz on one RTX 5090, a small MLP converts it to SMPL and GMR retargets it to the robot.
- Frequency-domain context. A conv encoder with a differentiable FFT turns each 1 s motion window into per-channel frequency, amplitude, offset and phase. Instead of a stacked preview, the actor gets ten frames of proprioception, one next-step target and codes covering about ten future frames.
- Extrapolation. A second encoder, trained for multi-step prediction, advances the latent phase to produce 50 Hz references between visual updates. It adds only the predicted increment to the latest measurement and rejects updates inconsistent with its motion model.
Results
- Ablation: the motion code plus one target reaches 99% success at 0.25 rad MPJPE. Ten explicit future frames reach 95% at 0.28 rad with an 11.1% larger input.
- Extrapolation error is 0.0655 rad joint RMSE vs 0.7561 rad for linear extrapolation.
- Reconstruction on EMDB: 117.54 mm MPJPE vs 123.45 mm for the GVHMR pipeline used by SONIC.
- Tracking an unseen dance from video in MuJoCo, MoViC has the lowest errors on all three metrics against SONIC, OmniH2O and TWIST retrained on the same data. On the real robot it performs the dance, including single-leg poses, a bimanual box transfer and a 139 s bag carry while stepping.
Limitations
References are only as good as reconstruction and retargeting, and phase extrapolation cannot stay accurate through long occlusions or abrupt motion changes. The camera is fixed, neither module sees objects or contact forces, and the real-robot results are qualitative demonstrations.
Videos
Figures



Abstract
Real-time whole-body imitation from monocular video requires both a robust universal tracker and high-quality real-time reconstruction. We present MoViC, a framework that uses live RGB video as a control signal for humanoid robots without wearable tracking devices. The framework combines monocular human motion reconstruction, robot retargeting, and a transformer-based behavior foundation model. The policy receives proprioceptive history, a single explicit next-step reference, and a compact frequency-domain motion representation that supplies temporal context without explicitly stacking future reference frames, which reduces the input dimension and network size. During deployment, our extrapolation method bridges the 10-Hz visual estimation with 50-Hz controller, enabling asynchronous visual processing and policy execution. Hardware experiments further demonstrate highly dynamic motion tracking and device-free teleoperation of bimanual and single-arm manipulation tasks from live monocular RGB input. Project page: https://movicproject.github.io/ .
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.
