Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots

Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, Shuran Song

★ Must read February 2024

In-the-wild cup arrangement outdoors, placing an espresso cup on its saucer.

TL;DR A hand-held GoPro gripper with a fisheye lens and side mirrors records demonstrations anywhere, and a latency-matched Diffusion Policy with relative actions transfers them directly to robots. Trained on 1400 in-the-wild demos, a cup-arrangement policy succeeds 71.7% of the time in unseen environments.

What they do

Hand-held grippers can collect demonstrations anywhere, but earlier systems only transferred simple grasping or quasi-static pick-and-place to robots. UMI pairs a 3D-printed hand-held gripper, with a GoPro as its only sensor, with a policy interface designed so the same data runs on different robots.

How it works

Hardware. A 155° fisheye lens gives visual context, and two side mirrors give implicit stereo, digitally reflected before training. Visual-inertial SLAM built on ORB-SLAM3 and the GoPro’s IMU recovers metric 6-DoF poses under fast motion. Fiducial markers track the finger width continuously. Parts cost $73 plus $298 for the camera.
Same view on the robot. Robots carry the same soft fingers and camera mount, so no camera-robot calibration is needed.
Latency matching. Observation streams are aligned to the slowest one (usually the camera) using measured latencies, and actions are sent early to cover robot and gripper execution latency.
Relative actions. Diffusion Policy outputs end-effector trajectories relative to the current pose. Bimanual policies also see the relative pose between the two grippers.

Results

Cup arrangement: 20/20, and 18/20 with the same checkpoint on a Franka FR2. Without the fisheye lens it scores 11/20, with absolute actions 5/20.
Dynamic tossing lands 105/120 objects (87.5%) in the right bin, or 57.5% without latency matching. Bimanual sweater folding succeeds 14/20 (6/20 without inter-gripper pose). Seven-step dish washing succeeds 14/20.
In the wild: 3 demonstrators collected 1400 demos at 30 locations in 12 person-hours. The policy succeeds 43/60 (71.7%) in two unseen environments, including unseen cups. Trained on lab data only, it scores 0%.
SLAM against motion capture: 6.1 mm and 3.5° mean error. For cup arrangement, data collection is more than 3x faster than spacemouse teleoperation.

Limitations

Each robot’s kinematic limits are handled only by filtering demonstrations. SLAM needs textured scenes. Collecting data is still slower than demonstrating with a bare hand (48% and 64% of hand speed on the two tasks measured), because the gripper is heavy and has fewer degrees of freedom.

Videos

Dynamic tossing, sorting balls and Duplo pieces into bins out of reach.

Bimanual sweater folding, with both arms grasping the hem together.

Figures

From Figure 1: Hand-held demonstrations collected in many environments transfer to robot policies for dynamic, precis... — From Figure 1: Hand-held demonstrations collected in many environments transfer to robot policies for dynamic, precise, bimanual and long-horizon tasks.

From Figure 2: Left, the hand-held gripper with a GoPro as its only sensor. Middle, the 155° fisheye view, with side ... — From Figure 2: Left, the hand-held gripper with a GoPro as its only sensor. Middle, the 155° fisheye view, with side mirrors giving implicit stereo. Right, the matching robot gripper and camera, which see nearly the same view.

From Figure 5: (a) Observation streams are synchronized using measured latencies. (b) The policy maps image, relative... — From Figure 5: (a) Observation streams are synchronized using measured latencies. (b) The policy maps image, relative end-effector pose and gripper width to a sequence of relative poses and gripper widths. (c) Commands are sent ahead of time to cover execution latency.

From Figure 9: (a) Trained on in-the-wild data, the policy generalizes to new environments and new cups. (b) Trained ... — From Figure 9: (a) Trained on in-the-wild data, the policy generalizes to new environments and new cups. (b) Trained only on narrow-domain data, it does not. (c) Quantitative results in the unseen environments.

Abstract

We present Universal Manipulation Interface (UMI) -- a data collection and policy learning framework that allows direct skill transfer from in-the-wild human demonstrations to deployable robot policies. UMI employs hand-held grippers coupled with careful interface design to enable portable, low-cost, and information-rich data collection for challenging bimanual and dynamic manipulation demonstrations. To facilitate deployable policy learning, UMI incorporates a carefully designed policy interface with inference-time latency matching and a relative-trajectory action representation. The resulting learned policies are hardware-agnostic and deployable across multiple robot platforms. Equipped with these features, UMI framework unlocks new robot manipulation capabilities, allowing zero-shot generalizable dynamic, bimanual, precise, and long-horizon behaviors, by only changing the training data for each task. We demonstrate UMI's versatility and efficacy with comprehensive real-world experiments, where policies learned via UMI zero-shot generalize to novel environments and objects when trained on diverse human demonstrations. UMI's hardware and software system is open-sourced at https://umi-gripper.github.io.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong