Solving Rubik's Cube with a Robot Hand

OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, Lei Zhang

Seminal October 2019

TL;DR A Shadow Hand policy and a vision state estimator are trained only in simulation with automatic domain randomization (ADR), which keeps widening the randomization ranges as the model improves. On the real hand the best policy completes a 43-move fair scramble in 20% of trials and half of it in 60%.

What they do

A Shadow Dexterous Hand solves a Rubik’s cube with control and vision models trained only in simulation. A standard solver plans the moves; the learned system senses the cube’s pose and face angles and executes each flip and quarter-turn. The key ingredient is automatic domain randomization (ADR): a curriculum that starts from one calibrated environment and widens the randomization ranges whenever the model performs well enough.

How it works

ADR. Each randomized parameter (friction, cube size, latency, observation noise, rendering) has a uniform range. Workers fix one parameter at a range boundary and evaluate; above 20 successes per episode the bound moves outward, below 10 it moves back.
Policy. PPO trains an LSTM policy with an asymmetric critic that sees privileged simulator state, for months on 64 V100 GPUs and 920 worker machines (about 13 thousand years of experience).
State estimation. A three-camera ResNet50 model trained on ADR-rendered images predicts cube pose and face angles; a corner cut from each center sticker makes face angles readable from one frame. Most experiments read face angles from a sensorized Giiker cube instead.

Results

Real Rubik’s cube (10 trials, fixed 43-subgoal scramble): the best ADR policy averages 26.8 successes, completing the full scramble in 20% of trials and half in 60%. Manual DR averages 1.8. With vision-only face angles the same policy averages 12.8 and never completes it.
Real block reorientation: mean successes rise with ADR entropy from 1.4 to 32.0, almost 2x the earlier baseline (18.8). Manual DR retrained on the current setup scores 42.5 in simulation but 2.7 on hardware.
ADR reaches high sim2sim performance faster than any fixed-randomization run.
Signs of meta-learning: time per flip falls by about 1.6 s during a trial, and LSTM states predict cube size with over 80% accuracy.

Limitations

Face angles from vision alone are much less reliable, so the headline result relies on a sensorized cube, and fingertips are tracked by motion capture. The policy sometimes gets stuck on face rotations and seems more likely to drop the cube early in a trial or after being stuck. Robustness to gloves, tied fingers and occlusion was only shown qualitatively.

Videos

Figures

From Figure 1: A five-fingered humanoid hand, trained with reinforcement learning and automatic domain randomization,... — From Figure 1: A five-fingered humanoid hand, trained with reinforcement learning and automatic domain randomization, solving a Rubik's cube.

From Figure 2: System overview. ADR generates a growing distribution of randomized simulations that trains both an LS... — From Figure 2: System overview. ADR generates a growing distribution of randomized simulations that trains both an LSTM control policy with RL and a CNN that predicts the cube's pose and face angles. On the real robot the CNN reads cube pose from 3 cameras, motion capture tracks the fingertips, and face angles come from the vision model or the sensorized Giiker cube.

From Figure 16: Sim2sim block reorientation performance over training. ADR, which starts with no randomization and ex... — From Figure 16: Sim2sim block reorientation performance over training. ADR, which starts with no randomization and expands it gradually, makes progress much faster than runs with randomization fixed at small, medium, large or XL entropy.

Abstract

We demonstrate that models trained only in simulation can be used to solve a manipulation problem of unprecedented complexity on a real robot. This is made possible by two key components: a novel algorithm, which we call automatic domain randomization (ADR) and a robot platform built for machine learning. ADR automatically generates a distribution over randomized environments of ever-increasing difficulty. Control policies and vision state estimators trained with ADR exhibit vastly improved sim2real transfer. For control policies, memory-augmented models trained on an ADR-generated distribution of environments show clear signs of emergent meta-learning at test time. The combination of ADR with our custom robot platform allows us to solve a Rubik's cube with a humanoid robot hand, which involves both control and state estimation problems. Videos summarizing our results are available: https://openai.com/blog/solving-rubiks-cube/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong