← Paper Digest

HandUMI: A Modular Hand-Worn Parallel-Jaw Interface for Robot-Free Bimanual Data Capture

Brik Meza, Leonardo Perez, Alvaro Mendoza-Li, Aniela Ubillus, Mitshell Ramos, Raul Bastidas, Mauricio Olivera, Bryan Bastidas

November 2026

Real PiPER arms grasp two corners of the towel and fold it in half.

TL;DR HandUMI is an open-source hand-worn parallel-jaw gripper, tracked by VR controllers, for recording bimanual demonstrations without a robot; a pair costs USD 557 with a Quest 3S. Trained only on 562 such demonstrations, ACT folds a towel on real PiPER arms in 12 of 20 attempts.

What they do

Robot-free rigs such as UMI let people record manipulation demonstrations without a robot, but the recorded gripper may not match the robot’s in contact geometry or opening motion. HandUMI is a hand-worn parallel-jaw interface whose swappable fingertips copy the gripper already installed on the target robot, so the robot keeps its own gripper. Recordings stay robot-agnostic and are retargeted to an arm afterwards. Hardware and software are open source (Apache-2.0).

How it works

  • Device. Printed finger cradles drive translating jaws through a crank. A 12-bit servo encoder measures jaw width and a fisheye camera records 1280x960 at 30 fps. Tips exist for AgileX PiPER, ARX X5, the TRLC-DK1 Dream Gripper, Trossen WidowX AI and the UMI soft finger. Parts cost USD 207.11 per pair.
  • Tracking. Instead of wrist-camera SLAM, Quest 3/3S or PICO 4 Ultra controllers supply 6-DoF poses, and a ChArUco board defines the table frame. Gated calibration checks the gripper range, camera intrinsics, table frame and each tip’s TCP.
  • Retargeting. Episodes store two controller poses and two jaw widths in LeRobot v3 format. PyRoki IK turns them into joint data for seven bimanual arms, and per-robot screening rejects episodes with IK error, self-collision or large base-joint swings.

Results

  • Three operators recorded 641 towel-folding episodes in about 5 h. 562 were kept (111.9 min); 79 were dropped because their PiPER retargeting swung a base joint past 60°.
  • Across the seven simulated arms, retargeting has a median position error of at most 1.2 mm and rejects at most 4.7% of episodes.
  • On real PiPER arms, trained only on HandUMI data: ACT on all 562 episodes folds the towel in 12/20 attempts and fine-tuned π0.5 in 7/20 (not statistically distinguishable, Fisher p = 0.20). ACT on the first 101 episodes and π0.5 without fine-tuning each succeed 0/20.

Limitations

Cross-embodiment evidence is simulated IK only. The real test is one task on one PiPER setup, 20 attempts per model judged live by the operator, with no failure analysis. Capture is limited to headset tracking and parallel jaws, and each new gripper needs new tips. Jaw accuracy and the gain from matched tips are unmeasured, and there is no direct comparison with YUBI, the closest prior device.

Videos

Teaser of the hand-worn interface and the bimanual arms its recordings are retargeted to.
An operator records with the hand-worn grippers and VR controllers, with the live camera view on screen.

Figures

From Figure 1: Left, robot-free capture with the hand-worn device (fisheye camera, servo encoder, swappable fingertip...
From Figure 1: Left, robot-free capture with the hand-worn device (fisheye camera, servo encoder, swappable fingertips), tracked by PICO 4 Ultra or Meta Quest controllers. Right, recorded trajectories retargeted in simulation to seven bimanual arms.
From Figure 2: The towel-folding task, with the two wrist views, the top view, and the stages of a rollout.
From Figure 2: The towel-folding task, with the two wrist views, the top view, and the stages of a rollout.
Abstract

HandUMI captures robot-free bimanual demonstrations with a hand-worn parallel-jaw interface. Swappable fingertips match the robot's installed gripper; Quest/PICO controllers track hands and encoders measure aperture. Robot-agnostic recordings are retargeted by inverse kinematics to seven simulated bimanual arms (median position error $\leq 1.2$ mm; $\leq 4.7$% of episodes rejected). On bimanual towel folding, 562 robot-free demonstrations train policies on real PiPER arms: ACT succeeds in 12/20 attempts and fine-tuned $π_{0.5}$ in 7/20; ACT on the first 101 and base $π_{0.5}$ each succeed in 0/20. Hardware and software are open source (Apache-2.0), and a pair costs USD 557 with Quest 3S, excluding the optional ZED Mini and host computer. Project website: https://murobotics.ai/research/handumi.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.