Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Social Humanoid: Socially Responsive Human-Humanoid Interaction

Vongani H. Maluleke, James Ni, Kie Horiuchi, Lea Wilken, Jiahui Lei, Evonne Ng, Roei Herzig, Jitendra Malik, Trevor Darrell, Angjoo Kanazawa

November 2026

Social Humanoid running live on a Unitree G1 with a human partner.

TL;DR A Unitree G1 watches a person through an RGB-D camera and generates a whole-body reaction (handshake, high-five, hug) with a diffusion model over motion tokens shared by humans and the humanoid. On the real robot it succeeds in 90-100% of 30 trials per interaction, with 308 ms median latency.

What they do

Social Humanoid makes a 29-DoF Unitree G1 react to a human partner in real time. A camera watches the person, a generative model produces the humanoid’s whole-body response and predicts the partner’s next motion (also when they are occluded), and NVIDIA’s SONIC tracking controller executes it. For training data they retarget human-human motion capture into a 260-hour Human-Humanoid Interaction (HHI) benchmark.

How it works

  • Adaptive-HHI retargeting. Pass I retargets one person to the G1 with OmniRetarget’s interaction-mesh SQP, keeping wrist and elbow distances to the partner. Pass II adapts the partner’s pose to the shorter robot instead of shrinking the partner.
  • Shared-codebook VQ-VAE. Separate human and humanoid encoders quantize into one 512-entry codebook; two alignment losses make paired poses map to the same code.
  • Diffusion Forcing Transformer. Autoregressively generates human and humanoid tokens, each noised independently, so the sampling schedule alone switches between joint generation, partner prediction and inpainting.
  • Deployment. ZED 2i stereo camera, Fast SAM 3D Body for the human pose, AprilTags plus odometry for the robot, SONIC at 50 Hz, most compute offboard.

Results

  • Generated humanoid motion: W-MPJPE 0.128 vs 0.545 for Ready-to-React and about 1.1 for retrieval baselines, which still win on FD and foot skating.
  • Retargeting: wrist error 0.50 vs 1.38 cm for OmniRetarget on CHI3D; downstream tracking success on Harmony4D 97% vs 81% (OmniRetarget) and 62% (PHC).
  • Real G1, 30 trials per scenario with 4 participants: 100% for handshake, wave, hug, bend and linked arms, 90% for high-five, matching simulation. Median camera-to-command latency is 308 ms.
  • The codebook also enables feed-forward shadow teleoperation (2.79 cm human-to-humanoid error).

Limitations

No hands, faces or objects. The system inherits SONIC’s and the pose estimator’s failure modes and tracks kinematics without modeling contact forces. Success rates measure execution, not interaction quality; there is no HRI study.

Videos

The G1 next to the reconstructed human and the predicted humanoid response.
A human-human interaction (left) and its retargeted humanoid and co-morphed partner (right).

Figures

From Figure 1: The G1 responding to a person with coordinated whole-body motion: a hug (top) and a greeting followed ...
From Figure 1: The G1 responding to a person with coordinated whole-body motion: a hug (top) and a greeting followed by posing together for a photo (bottom).
From Figure 2: Architecture. The human partner (red) is reconstructed from RGB-D video; human and humanoid states are...
From Figure 2: Architecture. The human partner (red) is reconstructed from RGB-D video; human and humanoid states are encoded into motion tokens by the VQ-VAE encoders, processed by the DFoT policy, and decoded into future human motion (blue) and humanoid motion (purple), which SONIC executes in sim or on the real robot.
From Figure 4: Adaptive-HHI Retargeting. Starting from two humans, the first pass retargets the ego human (orange) to...
From Figure 4: Adaptive-HHI Retargeting. Starting from two humans, the first pass retargets the ego human (orange) to the humanoid, preserving pose near the contact; the second pass co-morphs the partner's pose, gated by how much physical contact the interaction has.
From Figure 6: Real-world deployment, frames left to right in time. Top two rows: high-five and hand-wave trials. Bot...
From Figure 6: Real-world deployment, frames left to right in time. Top two rows: high-five and hand-wave trials. Bottom: one continuous take in which the humanoid goes from a handshake straight into a bend.
Abstract

Humanoid robots are typically trained as solitary agents, leaving them unable to coordinate physically with human partners who naturally anticipate and adjust to one another. To bridge this gap, we present Social Humanoid, a hierarchical system that generates and executes socially responsive whole-body motion on a 29-DoF Unitree G1 conditioned on a human partner's motion. Its high-level policy performs flexible sequence modeling of human and humanoid motion through a shared tokenized representation, generating reactive humanoid motion and anticipating the partner's motion when they are occluded, while a low-level whole-body tracking controller executes the generated motion on the robot. To train the system, we introduce a two-phase adaptive retargeting procedure that retargets one interaction partner to the robot while morphing the other partner to preserve contact dynamics, reachability, and relative body scale. We apply this procedure across five motion-capture datasets to produce a new Human-Humanoid Interaction (HHI) benchmark of 260 hours of human-humanoid interactions. Our results demonstrate that Social Humanoid produces coordinated, human-aligned behavior across diverse interaction types in simulation and the real world. Project page: https://von31.github.io/social-humanoid/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.