Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion

Zipeng Fu, Xuxin Cheng, Deepak Pathak

Seminal October 2022

TL;DR One RL policy controls both the legs and the arm of a low-cost quadruped with an arm mounted on top, so the legs bend and stretch to extend the arm's reach. In real-world cup picking it succeeds 80% of the time on easy and hard points, versus 30% and 10% for a built-in MPC plus arm IK baseline.

What they do

Legged manipulators are usually controlled hierarchically, with separate controllers for the arm and the legs. This paper trains one RL policy that outputs joint targets for all 12 leg joints and 6 arm joints of a Unitree Go1 carrying a WidowX 250s arm (about 6K USD of hardware), tracking an end-effector pose command and a base velocity command at the same time.

How it works

  • Unified policy. An MLP takes base, arm and leg state, the last action, both commands and a 20-D environment latent, and outputs PD joint targets at 50 Hz. It is trained with PPO in IsaacGym on rough terrain; the reward is a manipulation term plus a locomotion term, each with an energy penalty.
  • Advantage Mixing. Early on, arm actions are credited only with the manipulation advantage and leg actions only with the locomotion advantage; one parameter ramps linearly to the full sum. Without it the policy follows end-effector commands but never learns to walk.
  • Regularized Online Adaptation. A privileged-information encoder and an adaptation module that reads onboard observation history are trained together in one phase, with the encoder’s latent regularized toward the module’s estimate. RMA is the special case with no regularization and two phases.
  • Deployment. Runs onboard on a Raspberry Pi 4, commanded by joysticks, AprilTag visual feedback or open-loop replayed demonstrations.

Results

  • Simulation, out-of-distribution commands: end-effector error 0.63 vs 0.92 for separate arm and leg policies, survival 97.1% vs 92.0%. Arm workspace grows from 0.58 to 0.82 m³.
  • Against RMA, the latent gap drops from 0.31 to 2e-4 and end-effector error from 0.26 to 0.21, matching the privileged expert.
  • Real-world AprilTag pick-up (10 trials per task): 0.8 success on both easy and hard points in 5-5.6 s, vs 0.3 and 0.1 in 17-22 s for MPC+IK, which failed from IK singularities and self-collision.

Limitations

Object interaction is preliminary (picking, pressing, wiping). The gripper is opened and closed by joystick or a script, not the policy. Vision relies on AprilTags, and most failed picks came from the tag position not matching the cup.

Figures

From Figure 1: Whole-body control enlarges the workspace by bending and stretching the legs (left). Real-world tasks ...
From Figure 1: Whole-body control enlarges the workspace by bending and stretching the legs (left). Real-world tasks include wiping a whiteboard, picking up a cup, pressing door-open buttons, placing, throwing a cup into a bin and picking in clutter (right).
From Figure 2: In simulation a unified policy is trained conditioned on environment extrinsics; at deployment the ada...
From Figure 2: In simulation a unified policy is trained conditioned on environment extrinsics; at deployment the adaptation module estimates them, and the robot is commanded by teleoperation, vision or demonstration replay.
From Figure 4: With Advantage Mixing the velocity-tracking error drops sharply during training; without it, it stays ...
From Figure 4: With Advantage Mixing the velocity-tracking error drops sharply during training; without it, it stays high.
From Figure 6: The unified policy picks up a cup at easy and hard ground points with coordinated leg and arm motion.
From Figure 6: The unified policy picks up a cup at easy and hard ground points with coordinated leg and arm motion.
Abstract

An attached arm can significantly increase the applicability of legged robots to several mobile manipulation tasks that are not possible for the wheeled or tracked counterparts. The standard hierarchical control pipeline for such legged manipulators is to decouple the controller into that of manipulation and locomotion. However, this is ineffective. It requires immense engineering to support coordination between the arm and legs, and error can propagate across modules causing non-smooth unnatural motions. It is also biological implausible given evidence for strong motor synergies across limbs. In this work, we propose to learn a unified policy for whole-body control of a legged manipulator using reinforcement learning. We propose Regularized Online Adaptation to bridge the Sim2Real gap for high-DoF control, and Advantage Mixing exploiting the causal dependency in the action space to overcome local minima during training the whole-body system. We also present a simple design for a low-cost legged manipulator, and find that our unified policy can demonstrate dynamic and agile behaviors across several task setups. Videos are at https://maniploco.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.