Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation

Zipeng Fu, Tony Z. Zhao, Chelsea Finn

Seminal January 2024

The autonomous Cook Shrimp policy at 6x speed.

TL;DR A $32k bimanual mobile manipulator whose operator, tethered to the base at the waist, drives the wheels while puppeteering both arms, plus behavior cloning co-trained on existing static ALOHA data. With 20-50 demos per task, ACT reaches 80-95% success on six of seven household tasks.

What they do

Most imitation-learning results are on table-top tasks. Mobile ALOHA puts the low-cost ALOHA bimanual puppeteering rig on a wheeled base, so one person can teleoperate both arms and the base at the same time. Policies are then trained by plain behavior cloning. The paper also finds that co-training on existing static ALOHA data helps, even though that data covers unrelated tasks and a different arm layout.

How it works

Hardware. An AgileX Tracer base ($7,000) carries two leader and two follower arms, a 1.26 kWh battery, a laptop with an RTX 3070 Ti and three RGB webcams (two on the wrists, one on top). The whole system costs $32k.
Whole-body teleoperation. The operator is tethered to the base at the waist and backdrives the unpowered wheels (about 13 N of rolling resistance) while working the leader arms. Base velocities and arm joint positions are recorded together.
Policy. Actions are the 14 arm joint targets plus base linear and angular velocity, 16-D in total, so ACT, Diffusion Policy and VINN run almost unchanged. Arm and base actions within a chunk are offset to absorb the base’s velocity lag.
Co-training. Each batch samples equally from the task’s mobile demos and 825 static ALOHA demos, with zero-padded base actions and no other preprocessing.

Results

With co-trained ACT: Wipe Wine and Call Elevator 95%, Use Cabinet and High Five 85%, Rinse Pan and Push Chairs 80%. Cook Shrimp, a 75 s task with only 20 demos, reaches 40%.
Co-training helps most where precision is the bottleneck. Call Elevator goes from 0% to 95% (the button press rises from 5% to 100%) and Rinse Pan from 0% to 80%. Use Cabinet and High Five do not change.
On Wipe Wine, co-training scores 95% against 40% for pre-training on the static data and 50% for neither. With co-training, 35 demos beat 50 demos without it (70% vs 50%). Co-training also lifts Diffusion Policy (Wipe Wine 35% to 65%); results for VINN are mixed.
In a user study, 8 people who had never used the system cut their Use Cabinet time from 75 s to 36 s over five trials.

Limitations

The 90 cm x 135 cm footprint may not fit some narrow paths, and the fixed arm height makes low cabinets, ovens and dishwashers hard to reach. Policies are single-task, cannot improve on their own, and were trained only on demos from two expert operators.

Videos

Autonomous Call Elevator in real time, crossing the lobby toward the button.

Autonomous Use Cabinets in real time, putting a pot away in the wall cabinet.

Figures

From Figure 1: Mobile ALOHA hardware. Left: two wrist cameras, one top camera and onboard power and compute, with the... — From Figure 1: Mobile ALOHA hardware. Left: two wrist cameras, one top camera and onboard power and compute, with the operator tethered behind. Middle: with the teleoperation rig removed, the arms reach 65 cm to 200 cm high and 100 cm beyond the base. Right: technical specifications.

From Figure 2: Six of the seven tasks (Wipe Wine, Cook Shrimp, Wash Pan, Use Cabinet, Take Elevator, Push Chairs) wit... — From Figure 2: Six of the seven tasks (Wipe Wine, Cook Shrimp, Wash Pan, Use Cabinet, Take Elevator, Push Chairs) with their sub-steps and randomization, plus sketches of the base path for each.

From Figure 3: Wipe Wine success with 25, 35 and 50 Mobile ALOHA demos. Co-training with static ALOHA data is better ... — From Figure 3: Wipe Wine success with 25, 35 and 50 Mobile ALOHA demos. Co-training with static ALOHA data is better at every data size.

Abstract

Imitation learning from human demonstrations has shown impressive performance in robotics. However, most results focus on table-top manipulation, lacking the mobility and dexterity necessary for generally useful tasks. In this work, we develop a system for imitating mobile manipulation tasks that are bimanual and require whole-body control. We first present Mobile ALOHA, a low-cost and whole-body teleoperation system for data collection. It augments the ALOHA system with a mobile base, and a whole-body teleoperation interface. Using data collected with Mobile ALOHA, we then perform supervised behavior cloning and find that co-training with existing static ALOHA datasets boosts performance on mobile manipulation tasks. With 50 demonstrations for each task, co-training can increase success rates by up to 90%, allowing Mobile ALOHA to autonomously complete complex mobile manipulation tasks such as sauteing and serving a piece of shrimp, opening a two-door wall cabinet to store heavy cooking pots, calling and entering an elevator, and lightly rinsing a used pan using a kitchen faucet. Project website: https://mobile-aloha.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong