Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware

Tony Z. Zhao, Vikash Kumar, Sergey Levine, Chelsea Finn

★ Must read April 2023

Learned policy slotting a battery into a remote control, real time.

TL;DR ALOHA is a bimanual teleoperation rig built from off-the-shelf arms for under $20k, and ACT is an imitation learning policy that predicts chunks of future joint targets with a transformer CVAE. From 50 demonstrations per task, ACT slots a battery 96% of the time and opens a translucent condiment cup 84% of the time.

What they do

Fine bimanual tasks such as opening a condiment cup or slotting a battery need millimeter precision, while low-cost arms are accurate to only 5-8 mm. The paper pairs ALOHA, a teleoperation rig built from off-the-shelf ViperX arms and 3D-printed parts for under $20k, with ACT (Action Chunking with Transformers), an imitation learning policy that maps webcam images directly to joint targets and corrects errors from closed-loop visual feedback.

How it works

  • Teleoperation. The operator backdrives two smaller WidowX leader arms, and the ViperX followers copy their joint positions (no IK). Four 480x640 webcams (top, front, two wrists); teleoperation and recording run at 50 Hz.
  • Action chunking. The policy predicts the next k target joint positions for both arms instead of one step, shortening the effective horizon k-fold and handling pauses in demonstrations.
  • Temporal ensemble. The policy is queried every step, and the overlapping predictions for the same timestep are averaged with exponential weights for smoother motion.
  • CVAE. ACT is trained as a conditional VAE with a style variable z, set to zero at test time, to model noisy human demonstrations. About 80M parameters, trained from scratch per task in about 5 h on one RTX 2080 Ti.

Results

  • With 50 demonstrations per task (100 for Thread Velcro), final real-world success is 88% on Slide Ziploc, 96% Slot Battery, 84% Open Cup, 92% Put On Shoe, 64% Prep Tape and 20% Thread Velcro. Baselines make no progress past the first stage on the first two; BeT gets zero final success on the other four.
  • In simulation, ACT beats the best prior method by 59%, 49%, 29% and 20% across two tasks with scripted or human data.
  • Ablations: success rises from 1% at k=1 to 44% at k=100; without the CVAE, success on human data drops from 35.3% to 2%.
  • In a 6-person user study, teleoperating at 5 Hz instead of 50 Hz takes 62% longer.

Limitations

The arms lack the torque for high-force tasks, and the grippers cannot do tasks that need multiple fingers or fingernails. ACT failed to learn candy unwrapping (0/10 in the first evaluation) and opening a flat ziploc bag, and reached only 20% on Thread Velcro, where the black tie is hard to see.

Videos

Learned policy tipping over a condiment cup, lifting it and prying the lid open, real time.
Learned policy putting a shoe on a mannequin foot, real time.

Figures

From Figure 2: The ALOHA setup. Left: front, top and two wrist cameras around the bimanual workspace. Middle: the han...
From Figure 2: The ALOHA setup. Left: front, top and two wrist cameras around the bimanual workspace. Middle: the handle-and-scissor mechanism on the leader arm and the see-through custom grippers. Right: specs of the ViperX 6-DoF follower arm.
From Figure 3: ACT trained as a conditional VAE. Left: the encoder compresses joint positions and the action sequence...
From Figure 3: ACT trained as a conditional VAE. Left: the encoder compresses joint positions and the action sequence into a style variable z and is discarded at test time. Right: the policy fuses four camera images, joint positions and z with a transformer encoder and predicts an action sequence with a transformer decoder. Bottom (Figure 4): action chunking, and temporal ensembling of overlapping chunks.
From Figure 5: The six real-world tasks, with their initial setups and subtasks.
From Figure 5: The six real-world tasks, with their initial setups and subtasks.
From Figure 7: (a) Success rate against chunk size k for ACT and two baselines given action chunking. (b) Temporal en...
From Figure 7: (a) Success rate against chunk size k for ACT and two baselines given action chunking. (b) Temporal ensembling helps ACT and BC-ConvMLP but hurts VINN. (c) Removing the CVAE barely matters on scripted data but collapses success on human data. (d) User-study task times at 5 Hz and 50 Hz teleoperation.
Abstract

Fine manipulation tasks, such as threading cable ties or slotting a battery, are notoriously difficult for robots because they require precision, careful coordination of contact forces, and closed-loop visual feedback. Performing these tasks typically requires high-end robots, accurate sensors, or careful calibration, which can be expensive and difficult to set up. Can learning enable low-cost and imprecise hardware to perform these fine manipulation tasks? We present a low-cost system that performs end-to-end imitation learning directly from real demonstrations, collected with a custom teleoperation interface. Imitation learning, however, presents its own challenges, particularly in high-precision domains: errors in the policy can compound over time, and human demonstrations can be non-stationary. To address these challenges, we develop a simple yet novel algorithm, Action Chunking with Transformers (ACT), which learns a generative model over action sequences. ACT allows the robot to learn 6 difficult tasks in the real world, such as opening a translucent condiment cup and slotting a battery with 80-90% success, with only 10 minutes worth of demonstrations. Project website: https://tonyzhaozh.github.io/aloha/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.