Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, Sergey Levine

Seminal October 2019

MT10: the ten training tasks, from button press to window close.

TL;DR Meta-World is an open-source MuJoCo benchmark of 50 distinct Sawyer manipulation tasks with randomized object and goal positions, built to test multi-task and meta-RL on broad task distributions. Seven existing algorithms struggle on it; the best multi-task method reaches 68.3% success on 10 tasks and 38.5% on 50, and no meta-RL method solves more than 39.9% of held-out tasks.

What they do

Meta-RL methods have mostly been tested on narrow task families, such as different running directions for one simulated robot, which cannot show whether they adapt to genuinely new behaviors. Meta-World is an open-source benchmark of 50 qualitatively distinct table-top manipulation tasks for a simulated Sawyer arm, plus evaluation protocols of increasing difficulty. The authors run 7 multi-task and meta-RL algorithms on it.

How it works

Shared structure. Every task uses the same arm, the same action (3-D end-effector displacement plus a gripper torque) and a 39-D observation of gripper, object and goal positions. Tasks combine reaching, pushing and grasping on objects with different shapes and joints: doors, drawers, windows, faucets, pegs. Built on MuJoCo and Gym.
Parametric plus non-parametric variation. Each task randomizes object and goal positions, so a policy cannot tell a door from a drawer by where it sits, and distinct tasks can overlap (closing a drawer can look like pushing a block).
Rewards on one scale. Each task has a shaped multi-component reward (reach, grasp, place) between 0 and 10. Success is a task-specific distance threshold, such as 5 cm.
Protocols. ML1 adapts to new goals within one task. MT10 and MT50 train one task-conditioned policy on 10 or 50 tasks. ML10 and ML45 meta-train on 10 or 45 tasks and adapt to 5 held-out ones, with no task ID given.

Results

Single-task SAC learns all 50 tasks to some degree and PPO solves a large majority, so each task is solvable on its own.
MT10: multi-task SAC reaches 68.3% average success, MT-PPO 30.5%, MT-TRPO 31.3%, task embeddings 20.9%. On MT50 the best are MT-SAC at 38.5% and MT-PPO at 35.4%.
ML10 test tasks: RL² 35.8%, MAML 31.6%, PEARL 13%, although RL² reaches 86.9% on its training tasks. ML45 test tasks: MAML 39.9%, RL² 33.3%, PEARL 22%.
Meta-training success is also limited (MAML 44.4% on ML10, 40.7% on ML45), which the authors read as optimization being harder in the meta-learning setting.

Limitations

Policies see ground-truth object poses and get dense shaped rewards; image observations and sparse rewards are left to future work, as are long-horizon compositional tasks, infrequent resets and tasks that arrive over time. All results are in simulation.

Videos

ML45: the 45 meta-training tasks and the 5 held-out test tasks.

ML1: pick-and-place with a different goal location for each task.

Figures

From Figure 1: The 50 Meta-World manipulation tasks. In the hardest evaluation, ML45, experience from 45 training tas... — From Figure 1: The 50 Meta-World manipulation tasks. In the hardest evaluation, ML45, experience from 45 training tasks (left) is used to quickly learn distinctly new test tasks (right).

From Figure 2: Parametric versus non-parametric variation. All reach-puck tasks (left) differ only in puck position, ... — From Figure 2: Parametric versus non-parametric variation. All reach-puck tasks (left) differ only in puck position, while reach puck and open window (right) differ non-parametrically.

From Figure 3: Three evaluation protocols: adaptation within one task (ML1), multi-task training across 10 task famil... — From Figure 3: Three evaluation protocols: adaptation within one task (ML1), multi-task training across 10 task families (MT10), and adaptation to new tasks (ML10).

From Figure 5: Per-task success rates of four multi-task RL algorithms on MT10 over 10 seeds; MT-SAC performs best.

Abstract

Meta-reinforcement learning algorithms can enable robots to acquire new skills much more quickly, by leveraging prior experience to learn how to learn. However, much of the current research on meta-reinforcement learning focuses on task distributions that are very narrow. For example, a commonly used meta-reinforcement learning benchmark uses different running velocities for a simulated robot as different tasks. When policies are meta-trained on such narrow task distributions, they cannot possibly generalize to more quickly acquire entirely new tasks. Therefore, if the aim of these methods is to enable faster acquisition of entirely new behaviors, we must evaluate them on task distributions that are sufficiently broad to enable generalization to new behaviors. In this paper, we propose an open-source simulated benchmark for meta-reinforcement learning and multi-task learning consisting of 50 distinct robotic manipulation tasks. Our aim is to make it possible to develop algorithms that generalize to accelerate the acquisition of entirely new, held-out tasks. We evaluate 7 state-of-the-art meta-reinforcement learning and multi-task learning algorithms on these tasks. Surprisingly, while each task and its variations (e.g., with different object positions) can be learned with reasonable success, these algorithms struggle to learn with multiple tasks at the same time, even with as few as ten distinct training tasks. Our analysis and open-source environments pave the way for future research in multi-task learning and meta-learning that can enable meaningful generalization, thereby unlocking the full potential of these methods.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong