RoboMoCap: A Large-Scale Whole-Body Motion Capture Dataset with Egocentric Vision for Robot Learning

Mayur Deshmukh, Aaron Tung, Junyi Li, Nikash Bhardwaj, Ademi Adeniji, Vincent Liu, Michael Cho, Jiafei Duan

November 2026

Synchronized streams for one take: mocap skeleton and fitted body, third-person view and the head-mounted cameras.

TL;DR RoboMoCap is a dataset of over 200 hours from 14 people that pairs six head-mounted cameras and two wrist cameras with 120 Hz optical full-body motion capture, focused on locomotion and loco-manipulation. In its benchmarks the best egocentric pose method still misplaces feet by 85 mm, and retargeting to a Unitree G1 trades fidelity against feasibility.

What they do

Head-mounted cameras rarely see the wearer’s legs and feet, so egocentric datasets give weak supervision for locomotion. RoboMoCap records people in an optical motion-capture studio while they wear a six-camera headset and wrist cameras, so every egocentric frame comes with measured whole-body motion. It holds over 200 hours from 14 participants across 291 predefined tasks, and the authors benchmark existing methods on it.

How it works

Capture. A RoboCap headset (six synchronized 1080p cameras at 30 Hz, two IMUs), a RoboWrist camera on each wrist, one third-person camera, and a NOKOV system tracking 53 body markers at 120 Hz in a 7 x 7 m area. Scanned objects carry four markers for 6-DoF pose.
Alignment. Headset and mocap clocks are aligned by cross-correlating gyroscope angular speed with that of a marker cluster on the headset; camera poses come from a fixed transform to that cluster.
Annotation. SMPL-X is fit to the markers. Hand poses are vision-based estimates aligned to the SMPL-X wrists and are marked as estimated. Each task has a text description and episode timestamps.
Protocol. Eight action categories; locomotion primitives, posture and dynamics, and loco-manipulation make up about 80.6% of the hours.

Results

Quality: 99.81% valid skeleton frames and 4.78 mm median SMPL-X marker error.
Egocentric body reconstruction with pretrained models: EgoAllo has the lowest MPJPE (80.8 mm) and AGRoL the lowest world-frame error (95.4 mm).
Feet: EgoAllo is best with 90.5% contact F1, yet its footprint error is 85.1 mm and step displacement error 154.1 mm.
Retargeting to a 29-DoF Unitree G1 over 543 test sessions: OmniRetarget has the lowest global body error (34.8 mm against 90.7 mm for GMR) but 53.6% clip success; PHC reaches 79.6% and GMR 7.4%. All methods are near joint limits in over 70% of frames.

Limitations

Everything is recorded in one instrumented studio, which limits environments and terrain. Marker occlusions and body-model fitting errors can affect annotations, and finger motion is estimated from vision. The benchmarks stop at kinematic retargeting; validating the data through downstream humanoid execution is future work.

Videos

Third-person and fisheye egocentric views next to the reconstructed body motion while carrying a box.

A scanned box replayed along its optically tracked 6-DoF trajectory.

Figures

From Figure 1: RoboMoCap connects egocentric vision with whole-body motion for humanoid robot learning. (1) Egocentri... — From Figure 1: RoboMoCap connects egocentric vision with whole-body motion for humanoid robot learning. (1) Egocentric Vision captures synchronised multi-view observations from head-mounted, wrist-mounted, and exocentric cameras. (2) Human Motion provides optically captured whole-body motion, articulated hand poses, and tracked 3D objects. (3) Robot Learning leverages diverse locomotion, manipulation, and loco-manipulation demonstrations to support humanoid robot learning. The dataset comprises over 200 hours of recordings from 14 participants, covering over 290+ actions across eight categories.

From Figure 2: RoboMoCap hardware setup. (A) RoboCap for head-mounted egocentric sensing. (B) RoboWrist for wrist-mou... — From Figure 2: RoboMoCap hardware setup. (A) RoboCap for head-mounted egocentric sensing. (B) RoboWrist for wrist-mounted visual capture. (C) Motion capture suit with reflective markers for NOKOV optical tracking. (D) Recording studio equipped with NOKOV optical tracking cameras. (E) Example rigid object fitted with reflective markers for 6-DoF pose tracking.

From Figure 3: Overview of the RoboMoCap capture and annotation pipeline. (1) Capture: A multimodal recording system ... — From Figure 3: Overview of the RoboMoCap capture and annotation pipeline. (1) Capture: A multimodal recording system combines head-mounted and wrist-mounted cameras, third-person video, and NOKOV optical motion capture to record synchronised egocentric observations, full-body motion, and object interactions. (2) Alignment and fusion: Visual observations and optical motion capture are temporally aligned across 30 fps video and 120 Hz motion measurements. Local hand-pose estimates are aligned and fused with tracked wrist poses, while body joint trajectories and rigid-object poses are reconstructed from optical MoCap. (3) Representations: The resulting annotations provide calibrated camera observations, world-frame hand poses, SMPL-X body motion sequences, and tracked object motion for egocentric whole-body motion understanding and humanoid learning.

From Figure 4: Locomotion and loco-manipulation diversity in RoboMoCap. (A) Distributions of pelvis translational spe... — From Figure 4: Locomotion and loco-manipulation diversity in RoboMoCap. (A) Distributions of pelvis translational speed and turning rate across locomotion primitives, posture and dynamics, and loco-manipulation activities. (B) Step-width distributions across forward walking, turning, lateral/backward walking, and object carrying. (C) Cumulative distributions of angular disagreement between head orientation and travel direction across movement conditions, highlighting the challenges of inferring locomotion from egocentric observations.

Abstract

Humanoid robots must coordinate locomotion and manipulation to navigate complex environments and perform everyday tasks. Learning such behaviours from human demonstrations requires accurate whole-body motion paired with egocentric vision, particularly for lower-body movements and foot placements that are frequently invisible from a first-person perspective. However, existing egocentric datasets often lack high-fidelity optical motion capture or systematic coverage of diverse loco-manipulation behaviours. To address this gap, we introduce RoboMoCap, a large-scale multimodal dataset combining egocentric vision with marker-based optical motion capture, specifically designed for locomotion-centric humanoid learning. RoboMoCap comprises over 200 hours of recordings from 14 participants, featuring six head-mounted cameras, wrist-mounted visual and inertial sensing, and optical full-body motion capture at 120 Hz. The dataset systematically covers diverse locomotion primitives, dynamic whole-body movements, and loco-manipulation tasks, including variable-speed walking, turning, constrained navigation, terrain traversal, object transportation, and manipulation during locomotion. It provides temporally aligned multimodal observations and metric whole-body motion representations, enabling the study of human movement beyond directly observable body regions. We systematically evaluate existing methods on three benchmark tasks: egocentric full-body motion reconstruction, foot contact and placement estimation, and human-to-humanoid motion retargeting. By pairing egocentric vision with high-fidelity human motion across diverse locomotion and interaction scenarios, RoboMoCap provides a foundation for egocentric motion understanding and humanoid learning from human demonstrations.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong