Asymmetric Actor Critic for Image-Based Robot Learning

Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, Pieter Abbeel

Seminal October 2017

TL;DR Train an image-based policy in simulation with a critic that sees the full simulator state while the actor sees only RGBD images. With domain randomization, the policy transfers to a real Fetch robot with no real-world training data and succeeds 5/5 on picking, pushing and block moving.

What they do

Policies trained in simulation must run on camera images in the real world, but learning from images alone is slow because of high dimensionality and partial observability. The simulator, however, knows the full state. This paper trains an actor-critic in which only the critic gets the full state, while the actor (the deployed policy) gets rendered RGBD images. Combined with domain randomization, the image policy transfers to a real robot without any real-world data.

How it works

Asymmetric inputs. DDPG with Hindsight Experience Replay and sparse goal-reaching rewards. The critic is a fully connected network on full state, goal state and action; the actor is a CNN on the current image and an image of the goal. The full state is used only during training.
Bottleneck. An optional actor layer is trained with an L2 loss to predict the full state, which speeds up and stabilizes training.
Domain randomization. Textures, lighting, camera position, orientation and focal length are randomized, and noise is added to depth.

Results

In simulation (Particle, Reacher, Fetch Pick), asymmetric DDPG and HER beat their symmetric (image-only critic) versions. A DAgger behaviour-cloning baseline from a full-state expert learns faster at first but plateaus below asymmetric HER.
On a real Fetch arm, asymmetric HER succeeds 5/5 on Pick, Forward Push and Block Move. Asymmetric DDPG and symmetric HER succeed 0/5 on all three; DAgger gets 3/5 on Pick only.
Without randomization, policies fail on the real robot (0/5 on every task); without viewpoint randomization only Block Move works (4/5).

Limitations

The paper does not discuss limitations. Real-robot tests are 5 trials per task, each moving a single block. In Fetch Pick the gripper rotation is fixed, and half of the training episodes start from a recorded grasped state to ease exploration.

Figures

From Figure 1: Policies trained in randomized simulated environments with an asymmetric actor critic are tested on a ... — From Figure 1: Policies trained in randomized simulated environments with an asymmetric actor critic are tested on a real robot without any real-world training data.

From Figure 2: The critic gets full states and full goal states; the actor gets partial observations of the scene and... — From Figure 2: The critic gets full states and full goal states; the actor gets partial observations of the scene and of the goal.

From Figure 5: Asymmetric inputs outperform symmetric inputs on Particle, Reacher and Fetch Pick; the behaviour cloni... — From Figure 5: Asymmetric inputs outperform symmetric inputs on Particle, Reacher and Fetch Pick; the behaviour cloning baseline (dashed) learns faster at first but saturates below asymmetric HER. Shaded regions are one standard deviation over 5 seeds.

From Figure 7: Successive frames of asymmetric HER policies on the real Pick, Forward Push and Block Move tasks.

Abstract

Deep reinforcement learning (RL) has proven a powerful technique in many sequential decision making domains. However, Robotics poses many challenges for RL, most notably training on a physical system can be expensive and dangerous, which has sparked significant interest in learning control policies using a physics simulator. While several recent works have shown promising results in transferring policies trained in simulation to the real world, they often do not fully utilize the advantage of working with a simulator. In this work, we exploit the full state observability in the simulator to train better policies which take as input only partial observations (RGBD images). We do this by employing an actor-critic training algorithm in which the critic is trained on full states while the actor (or policy) gets rendered images as input. We show experimentally on a range of simulated tasks that using these asymmetric inputs significantly improves performance. Finally, we combine this method with domain randomization and show real robot experiments for several tasks like picking, pushing, and moving a block. We achieve this simulation to real world transfer without training on any real world data.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong