Residual Reinforcement Learning for Robot Control

Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, Sergey Levine

Seminal December 2018

TL;DR Add an RL policy on top of a hand-engineered feedback controller and train only that residual, directly on a real Sawyer arm. When the blocks in an insertion task are randomly tilted, residual RL succeeds in 15/20 trials versus 2/20 for the hand-engineered controller, after about three hours of real-world training.

What they do

Conventional feedback controllers struggle with contact-rich manufacturing tasks because contacts and friction are hard to model, so they need manual tuning. RL can handle contacts, but learning from scratch needs a lot of real-world interaction, which can be unsafe. Residual RL splits the job: a hand-designed controller handles the robot’s motion, and an RL policy learns an additive correction that handles contacts and the objects. They test it on block insertion with a Sawyer arm, both in MuJoCo and on hardware.

How it works

Additive control. The command is u = π_H(s_m) + π_θ(s_m, s_o): a fixed hand-engineered controller acting on the robot state, plus a learned policy that also sees the object state. The reward splits the same way. One term rewards moving the gripper between the blocks; the other rewards keeping the standing blocks upright and in place.
Learning. The residual policy is trained off-policy with TD3 (the rlkit implementation). Exploration noise is added only to the learned part.
Real-world setup. A compliant joint-space impedance controller drives the arm. Block poses come from a camera-based tracking system, and the observation includes only the z-axis end-effector force.
Sim-to-real variant. With the side blocks fixed in place, the base controller is a policy trained with deep RL in simulation, and residual RL refines it on the real robot.

Results

In simulation and on hardware, residual RL learns faster than RL alone and reaches better final performance.
When each block is randomly tilted ±20° at reset, residual RL succeeds in 15/20 trials and the hand-engineered controller in 2/20. With aligned blocks, both succeed in 20/20. Learning took about 8,000 samples, roughly three hours on the real robot.
In simulation, the hand-engineered controller’s return falls as block rotation, control noise or control bias increases. Residual RL’s return stays roughly constant across all three.
Initialized from the simulation policy, residual RL solves the fixed-block task in under 1,000 timesteps of real interaction.

Limitations

The real-world system needs a carefully crafted vision setup to estimate block positions and angles. This limits it to known scenes and can drop information the controller needs, such as precise edge locations. All experiments use a single block-insertion task.

Videos

Figures

From Figure 1: A real-world rollout of residual RL. The policy adapts to the orientation of the standing blocks and i... — From Figure 1: A real-world rollout of residual RL. The policy adapts to the orientation of the standing blocks and inserts the held block between them.

From Figure 2: The real-world block assembly task. The robot must insert a block between the two blocks on the table ... — From Figure 2: The real-world block assembly task. The robot must insert a block between the two blocks on the table without moving them or tipping them over.

From Figure 3: Real-world success rate during training. On every reset, each block is set straight or tilted 20 degre... — From Figure 3: Real-world success rate during training. On every reset, each block is set straight or tilted 20 degrees clockwise or counterclockwise. Residual RL is compared with the hand-engineered controller.

From Figure 5: Real-world block insertion for sim-to-real transfer. "Sim" means the real-world policy was initialized by RL in simulation. Residual RL with that initialization solves the task with little real experience.

Abstract

Conventional feedback control methods can solve various types of robot control problems very efficiently by capturing the structure with explicit models, such as rigid body equations of motion. However, many control problems in modern manufacturing deal with contacts and friction, which are difficult to capture with first-order physical modeling. Hence, applying control design methodologies to these kinds of problems often results in brittle and inaccurate controllers, which have to be manually tuned for deployment. Reinforcement learning (RL) methods have been demonstrated to be capable of learning continuous robot controllers from interactions with the environment, even for problems that include friction and contacts. In this paper, we study how we can solve difficult control problems in the real world by decomposing them into a part that is solved efficiently by conventional feedback control methods, and the residual which is solved with RL. The final control policy is a superposition of both control signals. We demonstrate our approach by training an agent to successfully perform a real-world block assembly task involving contacts and unstable objects.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong