Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation

Mohit Shridhar, Lucas Manuelli, Dieter Fox

Seminal September 2022

Real Franka puts a tape roll, then a marker, into the top drawer.

TL;DR PerAct encodes a language goal and a voxelized RGB-D scene with a Perceiver Transformer and picks the next gripper pose by classifying voxels. One agent trained on 18 RLBench tasks beats a 3D ConvNet baseline by 2.83x with 100 demos per task, and a real Franka learns 7 tasks from 53 demos.

What they do

Behavior cloning that maps 2D images straight to 6-DoF actions needs a lot of data. PerAct (Perceiver-Actor) puts both observation and action in a 3D voxel grid and treats each step as classification: given a language goal and the current scene, pick the voxel the gripper should move to, plus its rotation and gripper state. One Transformer learns 18 RLBench tasks (249 variations); another learns 7 real-world tasks.

How it works

Keyframes and voxels. Demonstrations are reduced to keyframes where the arm comes to rest. RGB-D views are fused into a 100³ grid over 1 m³; rotation uses 5° bins per axis, plus binary gripper-open and collision-avoidance flags.
Perceiver encoder. 5³ voxel patches (8000 tokens) and CLIP language tokens are encoded by a Perceiver whose 2048 latents cross-attend to the input, avoiding quadratic self-attention over the full sequence.
Action heads. Per-voxel features form a 3D Q-map for translation; pooled features feed heads for rotation, gripper and collision. Training is cross-entropy on expert keyframes, with translation and yaw augmentation applied to voxels and actions together.
Execution. A motion planner reaches each predicted pose, then the agent observes again and acts until the goal is reached.

Results

Across 18 simulated tasks, PerAct beats the 3D ConvNet baseline C2FARM-BC in 25 of 36 evaluations, by 1.33x on average with 10 demos per task and 2.83x with 100. Image-to-action baselines are near zero on most tasks.
On open drawer, whose three handles look identical, PerAct exceeds 70% success while every C2FARM-BC variant stays near chance (about 33%).
Random or fixed-interval keyframes give zero success.
A real Franka with one Kinect-2, trained from scratch on 53 demos, presses a hand sanitizer 9/10 times, reaches 40-70% on five other tasks, and sweeps beans in 1 of 5 trials.

Limitations

Insert peg, stack cups and place cups stay near zero in the multi-task setting. Discrete poses executed by a sampling-based planner rule out dynamic and multi-finger manipulation. The agent sees only the current observation, train and test objects are the same, and the simulation agent took 16 days to train on 8 V100s.

Videos

Real Franka sweeps beans onto the gray dustpan.

Real Franka presses a hand sanitizer as a person moves it around the table.

Figures

From Figure 1: One language-conditioned agent for many 6-DoF tasks: 10 of the 18 simulated RLBench tasks (a-j) and 5 ... — From Figure 1: One language-conditioned agent for many 6-DoF tasks: 10 of the 18 simulated RLBench tasks (a-j) and 5 of the 7 real-world Franka Panda tasks (k-o), the latter learned from 53 demonstrations in total.

From Figure 2: A voxel grid built from RGB-D images is split into 3D patches and appended to the encoded language goa... — From Figure 2: A voxel grid built from RGB-D images is split into 3D patches and appended to the encoded language goal. A Perceiver Transformer encodes the sequence through a small set of latents, and a decoder predicts discretized translation, rotation, gripper-open and collision-avoidance actions, which a motion planner executes.

From Figure 5: Translation Q-predictions from PerAct, with the expert actions circled. The left two are simulated tas... — From Figure 5: Translation Q-predictions from PerAct, with the expert actions circled. The left two are simulated tasks, the right two real-world tasks.

From Figure 4: Success rate on open drawer over training steps for PerAct and C2FARM-BC with several voxelization sch... — From Figure 4: Success rate on open drawer over training steps for PerAct and C2FARM-BC with several voxelization schemes, all trained with 100 demonstrations.

Abstract

Transformers have revolutionized vision and natural language processing with their ability to scale with large datasets. But in robotic manipulation, data is both limited and expensive. Can manipulation still benefit from Transformers with the right problem formulation? We investigate this question with PerAct, a language-conditioned behavior-cloning agent for multi-task 6-DoF manipulation. PerAct encodes language goals and RGB-D voxel observations with a Perceiver Transformer, and outputs discretized actions by ``detecting the next best voxel action''. Unlike frameworks that operate on 2D images, the voxelized 3D observation and action space provides a strong structural prior for efficiently learning 6-DoF actions. With this formulation, we train a single multi-task Transformer for 18 RLBench tasks (with 249 variations) and 7 real-world tasks (with 18 variations) from just a few demonstrations per task. Our results show that PerAct significantly outperforms unstructured image-to-action agents and 3D ConvNet baselines for a wide range of tabletop tasks.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong