From a Single RGB Image to Robot Execution: Metric Scene Alignment and Contact-Guided Retargeting

Zhenxuan Ding, Yu Huang, Xinmiao Wang, Chenghao Sun, Ziyang Dong, Wenhao Cui

November 2026

TL;DR Turns one RGB photo of a scene into a reference motion that a Unitree G1 can execute. The pipeline reconstructs the scene, generates a human interaction in it, and fixes contact errors both before and after retargeting to the robot. It succeeds in 89% of simulated trials and 35 of 50 real-robot trials on boxes, steps, gaps and slopes.

What they do

Generating humanoid motions that interact with a scene, such as climbing onto a box, usually needs a prebuilt 3D model of that scene. This pipeline starts from a single RGB image instead. It reconstructs the scene, generates a human interaction in it, and retargets the motion to a Unitree G1. Two refinement stages fix the geometric errors that appear along the way, and the resulting reference can be tracked by existing humanoid motion trackers without scene-specific fine-tuning.

How it works

Scene and motion. SAM 3D Objects reconstructs metric scene geometry from the image, which is registered to the robot frame; a fixed generator produces a human interaction conditioned on that scene.
JMIR (before retargeting). Optimizes a residual on the human motion that pulls intended contact points onto the scene surface and penalizes penetration, while keeping human-scene offsets, residual size and smoothness close to the generated motion.
CGRR (after retargeting with GMR). Frame-by-frame trust-region SQP applies a bounded joint correction so that feet in contact sit on a support height field built from the scene, under joint-limit and collision constraints.
The final reference is passed to pretrained trackers (MimicLite, GMT, SONIC, Humanoid-GPT).

Results

Against GRAIL with the same input, refinement takes 87 s instead of 480 s, penetration drops from 1.59% to 1.23%, and InterMimic success rises from 63.5% to 84.3%.
Simulation, 100 trials per variant: GMR alone gives 0.57 cm support error and 75% success; the full pipeline gives 0.21 cm and 89%.
Real G1, 10 trials per terrain: 9/10 and 8/10 on two box heights (Box10, Box20), 6/10 on steps, 4/10 across a gap and 8/10 on a slope, 35/50 overall. Across trackers without fine-tuning, success is 50-70%; fine-tuning SONIC for 30 h reaches 90%.

Limitations

Only support-dominant interactions are covered, not manipulation of moving objects or changing contacts. Refinement is kinematic, without dynamic feasibility, and ignores uncertainty in the reconstructed scene. Real success (70%) trails simulation (89%), and the gap terrain succeeds only 4 of 10 times.

Figures

From Figure 1: Overview of the single-image-to-humanoid workflow. From a single RGB image, the pipeline reconstructs ... — From Figure 1: Overview of the single-image-to-humanoid workflow. From a single RGB image, the pipeline reconstructs a registered metric scene and generates a scene-conditioned human interaction. JMIR corrects human-scene inconsistency, GMR transfers the refined motion to the Unitree G1, and CGRR corrects the remaining robot-scene support mismatch to produce an executable robot reference.

From Figure 2: Two-stage geometric refinement across embodiment transfer. JMIR corrects the human-scene mismatch whil... — From Figure 2: Two-stage geometric refinement across embodiment transfer. JMIR corrects the human-scene mismatch while preserving the generated interaction structure. After GMR retargeting, CGRR applies a bounded, contact-guided correction to restore the robot's support contacts.

From Figure 5: One interaction from generation to execution. Rows show the generated human interaction, the refined r... — From Figure 5: One interaction from generation to execution. Rows show the generated human interaction, the refined robot reference after retargeting, and the physical G1 execution; columns are successive frames.

From Figure 6: Real-robot success per terrain on the Unitree G1, 10 trials each; the dashed line is the overall 70% a... — From Figure 6: Real-robot success per terrain on the Unitree G1, 10 trials each; the dashed line is the overall 70% across 50 trials.

Abstract

Extending humanoid interaction generation to new environments remains limited by the need for prebuilt 3D scene assets. Reconstructing scenes directly from single RGB images offers a way to overcome this limitation, but introduces a key challenge in maintaining geometric consistency throughout motion generation and transfer. Visually plausible human motions may be misaligned with the reconstructed scene, while retargeting can introduce additional support errors due to differences between human and robot embodiments. We present a unified pipeline that transforms a single RGB image into a humanoid motion reference. It reconstructs scene geometry from the input image, generates a human interaction conditioned on the reconstructed scene, and retargets the motion to a humanoid robot. To maintain geometric consistency throughout this process, we incorporate two complementary refinement stages before and after embodiment transfer. Before retargeting, Joint Metric Interaction Refinement (JMIR) refines human motion alignment with the registered metric scene while preserving the generated interaction structure. Following retargeting, Contact Guided Residual Retargeting (CGRR) corrects robot specific support errors to restore the intended support relationships. Together, these stages produce motion references that are consistent with the reconstructed scene and can be followed by multiple pretrained humanoid trackers without scene specific policy fine tuning. Compared with GRAIL, our method reduces refinement time from 480 s to 87 s, yielding a speedup of 5.5 times. The complete pipeline achieves a success rate of 89% in simulation and 70% across 50 physical trials on the Unitree G1, demonstrating its effectiveness in generating executable humanoid motion references from single RGB images.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong