ReGATE: Recurrent Gated Visual Fusion for Reactive Humanoid-Object Interaction

Hao Zhu, Jorge Peña-Queralta

November 2026

TL;DR ReGATE lets a Unitree G1 humanoid pick up, carry and release a box on user command using only onboard depth and proprioception, with learned gates that decide how much to trust vision. In simulation it acquires static boxes 85.2% of the time vs 48.5% for VisualMimic, and on hardware it still grabs a box after the camera is blacked out.

What they do

ReGATE is a whole-body controller that lets a Unitree G1 humanoid pick up, carry and put down a box using only proprioception and a head-mounted depth camera, with no motion capture or object-pose estimator. A user steers it online with planar velocity, yaw rate and a binary contact command, and it keeps working when the depth view is occluded or blacked out.

How it works

Teacher-student training in Isaac Sim. A PPO teacher with privileged object state and reference motions supplies a target latent and actions; the student reproduces them from 64x36 depth, proprioception and the command, then is refined with PPO.
Recurrent gated fusion. A CNN and ConvGRU turn depth into spatial tokens. Proprioception-conditioned cross-attention retrieves visual evidence, and head-wise and global gates set how much of it updates an always-on proprioceptive pathway. Auxiliary heads predict object pose, a posed object point cloud and end-effector contact.
Depth curriculum. The gates are held open on clean depth first, then occlusion and full-frame blackout are added, so the gates do not shut vision out before it becomes useful.
Phase-aware initialization. Resets sample approach, pre-contact, interact, pre-release and release phases with set probabilities, oversampling brief contact transitions.

Results

Static boxes at 11 lateral offsets over ±0.5 m in MuJoCo (box position not randomized in training): 85.2% success over 330 trials vs 48.5% for VisualMimic, higher at every offset; 100% for offsets from -0.2 to +0.1 m.
Boxes moving at 0.1-0.5 m/s: 85.9% over 440 trials.
Commanded sequences: all ten trials complete grasp, transport, release and return to standing.
On the G1 it grasps boxes at different lateral positions and picks up a box after the camera is blacked out before the contact command (qualitative).

Limitations

The approach to the box and the timing of the contact command are left to a higher-level controller and not evaluated. Hardware results are qualitative and cover one box task. Success drops to 63.3-76.7% at offsets of +0.2 to +0.4 m. The six wrist joints are held fixed. The gates are learned coefficients, not calibrated estimates of sensor reliability.

Figures

From Figure 1: ReGATE on a Unitree G1, frames left to right. (a) Turning the contact command on starts grasping and l... — From Figure 1: ReGATE on a Unitree G1, frames left to right. (a) Turning the contact command on starts grasping and lifting; turning it off makes the robot place and release the box. (b) After grasping, the robot follows planar velocity commands to carry the box while keeping balance and contact.

From Figure 2: Overview. (a) Observations available to the teacher and the student. (b) The privileged teacher produc... — From Figure 2: Overview. (a) Observations available to the teacher and the student. (b) The privileged teacher produces the target latent and teacher actions. (c) The student combines proprioception, commands and recurrent depth tokens to estimate that latent and output joint targets. Dashed arrows are training-only supervision.

From Figure 7: The real G1 picks up the box even when its vision input is completely blocked: perceive, blackout, con... — From Figure 7: The real G1 picks up the box even when its vision input is completely blocked: perceive, blackout, contact, grasp, stand.

Abstract

Humanoid loco-manipulation requires coordinating object acquisition, transport, and release from onboard sensing while responding to user commands and incomplete visual observations. We present ReGATE, a whole-body controller that combines egocentric depth, proprioception, and online locomotion and contact commands. Built on a teacher-student framework, ReGATE integrates interaction-state estimation through a recurrent adapter that predicts object geometry and physical contact without an external object-pose estimator. Head-wise and global gates regulate visual updates to a proprioceptive pathway. A visual-grounding and corruption curriculum trains the adapter, while phase-aware initialization increases coverage of brief contact transitions. In simulation, ReGATE achieves 85.2% success across 330 static-object acquisition trials, compared with 48.5% for VisualMimic under the same protocol, and 85.9% across 440 moving-object acquisition trials. Deployment on a Unitree G1 demonstrates commanded grasp-transport-release sequences and acquisition of laterally displaced boxes. A qualitative hardware experiment further demonstrates successful box acquisition when the camera is occluded after an initial observation and before the contact command. To the best of our knowledge, this is the first work to extend recurrent gated fusion of onboard depth and proprioception from perceptive locomotion to humanoid loco-manipulation under interrupted visual sensing.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong