← Paper Digest

MoViC: Real-Time Humanoid Whole-Body Control from Monocular Video Streams

Yimin Han, Jiahui Zhang, Wanyue Li, Zeren Luo, Yeke Chen, Liu Zhao, Peizhuo Li, Xiaoyu Ji, Shihao Dong, Xuechao Chen, Zhangguo Yu, Annan Tang, Peng Lu

November 2026

The humanoid mirrors an operator's whole-body motion live from a single webcam.

TL;DR MoViC drives a full-size humanoid from a single webcam: the operator is reconstructed at about 10 Hz, a learned extrapolator fills in 50 Hz references, and a transformer tracker conditioned on a compact frequency-domain motion code follows them. The tracker reaches 99% success versus 95% with a larger ten-frame preview, and the real robot carries a box and a bag under webcam teleoperation with no wearables.

What they do

Humanoid whole-body teleoperation usually needs motion capture or a VR headset. MoViC needs only a fixed RGB webcam. It reconstructs the operator’s body per frame, retargets the motion to the 24-DoF EngineAI PM01 and feeds it to a universal motion tracker trained with PPO on retargeted human motion datasets. The same policy tracks dance and teleoperates object manipulation with no task-specific training.

How it works

  • Video to reference. YOLO11-pose finds the person, SAM 3D Body reconstructs the mesh at about 10 Hz on one RTX 5090, a small MLP converts it to SMPL and GMR retargets it to the robot.
  • Frequency-domain context. A conv encoder with a differentiable FFT turns each 1 s motion window into per-channel frequency, amplitude, offset and phase. Instead of a stacked preview, the actor gets ten frames of proprioception, one next-step target and codes covering about ten future frames.
  • Extrapolation. A second encoder, trained for multi-step prediction, advances the latent phase to produce 50 Hz references between visual updates. It adds only the predicted increment to the latest measurement and rejects updates inconsistent with its motion model.

Results

  • Ablation: the motion code plus one target reaches 99% success at 0.25 rad MPJPE. Ten explicit future frames reach 95% at 0.28 rad with an 11.1% larger input.
  • Extrapolation error is 0.0655 rad joint RMSE vs 0.7561 rad for linear extrapolation.
  • Reconstruction on EMDB: 117.54 mm MPJPE vs 123.45 mm for the GVHMR pipeline used by SONIC.
  • Tracking an unseen dance from video in MuJoCo, MoViC has the lowest errors on all three metrics against SONIC, OmniH2O and TWIST retrained on the same data. On the real robot it performs the dance, including single-leg poses, a bimanual box transfer and a 139 s bag carry while stepping.

Limitations

References are only as good as reconstruction and retargeting, and phase extrapolation cannot stay accurate through long occlusions or abrupt motion changes. The camera is fixed, neither module sees objects or contact forces, and the real-robot results are qualitative demonstrations.

Videos

Webcam teleoperation of a bimanual box pick, reorient and place.
Webcam teleoperation of a single-arm bag carry.
Full video (YouTube)

Figures

From Figure 2: (a) A frozen encoder maps reference motion windows to frequency-domain codes. (b) The transformer acto...
From Figure 2: (a) A frozen encoder maps reference motion windows to frequency-domain codes. (b) The transformer actor combines proprioceptive history, one next-step target and the theta tokens, and outputs 24 joint position targets for PD control. (c) At deployment, SAM 3D Body reconstruction, extrapolation and GMR retargeting turn a roughly 10 Hz video stream into 50 Hz references. (d) PPO training with a privileged critic.
From Figure 8: Joint position, joint velocity and keypoint errors of MoViC, SONIC, OmniH2O and TWIST tracking the uns...
From Figure 8: Joint position, joint velocity and keypoint errors of MoViC, SONIC, OmniH2O and TWIST tracking the unseen Bencaogangmu dance from video; lower is better.
From Figure 10: Teleoperation. (a) Bimanual pick, reorient and place of a box. (b) Single-arm bag pick, carry and pla...
From Figure 10: Teleoperation. (a) Bimanual pick, reorient and place of a box. (b) Single-arm bag pick, carry and place in a 139 s continuous session. Insets show the operator, who wears no tracking device.
Abstract

Real-time whole-body imitation from monocular video requires both a robust universal tracker and high-quality real-time reconstruction. We present MoViC, a framework that uses live RGB video as a control signal for humanoid robots without wearable tracking devices. The framework combines monocular human motion reconstruction, robot retargeting, and a transformer-based behavior foundation model. The policy receives proprioceptive history, a single explicit next-step reference, and a compact frequency-domain motion representation that supplies temporal context without explicitly stacking future reference frames, which reduces the input dimension and network size. During deployment, our extrapolation method bridges the 10-Hz visual estimation with 50-Hz controller, enabling asynchronous visual processing and policy execution. Hardware experiments further demonstrate highly dynamic motion tracking and device-free teleoperation of bimanual and single-arm manipulation tasks from live monocular RGB input. Project page: https://movicproject.github.io/ .

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.