Robot-Matched Soft Fingers for UMI Data Collection: Lessons from 100 Hours of Pick and Place

Kobi Kelemen, Tomasz Niedziałkowski, Marcin Paszczak, Victoriano Montesinos

November 2026

R1 Pro picks the held-out soup into the basket after mid-training and fine-tuning.

TL;DR A battery-powered handheld UMI with soft, angled fingers that match the robot's gripper, used to collect 99.7 hours of pick and place over 53 objects. Mid-training π0.5 on this data raises success after fine-tuning on unseen objects from 35% to 57% on a Galaxea R1 Pro.

What they do

Handheld Universal Manipulation Interfaces (UMIs) collect demonstrations without a robot, but existing ones need external base stations, use rigid fingers that give no force signal, or use parallel jaws that limit grip. This paper builds on the YUBI interface: no base stations, battery-powered, and soft angled fingers that match the robot’s gripper and deform visibly under load as a visual proxy for force. With it the authors collect 99.7 hours (over 51,000 episodes) of single-object pick and place over 53 objects and test what the data does for a π0.5 policy.

How it works

Hardware. Soft TPU fingertips, 1.5x longer than YUBI’s, open up to 90°. A 450 g aluminum robot gripper shares their dimensions, fingertips and camera placement, so demonstrations map one-to-one. Meta Quest 3S controllers give the gripper poses; the headset, worn around the neck, carries a top camera. A full set costs about 3,000 USD and runs about 19 hours per charge.
Frame-free state and actions. The state is the left gripper’s pose in the right gripper’s frame, both apertures and the gravity direction in each gripper frame; actions are 30-step chunks relative to the current pose. Nothing depends on the tracking frame, so no calibration to the robot is needed.
Training. π0.5 is mid-trained on 92 hours over 49 objects from 5 operators, then fine-tuned on one held-out object at a time with 1.5-2.4 hours of data.

Results

50 trials per policy and object.

A top camera, present in only 48 of the 92 hours and dropped out half the time in training, raises mean success on 5 seen objects from 47% to 60%.
On 4 held-out objects on the R1 Pro, mid-training raises success after fine-tuning from 35% to 57% (45% to 74% on box, porridge and soup). Both fail on a wire that needs a precise grasp (4% and 6%). Without fine-tuning, the mid-trained policy succeeds in 7% of trials.
On the R1 Lite, which cannot reach 83% of the mid-training episodes and is fine-tuned on teleoperation data, mid-training still raises success from 80% to 89%.

Limitations

The handhelds are cabled to a battery and laptop, and tracking degrades when the grippers leave the headset cameras’ view. Evaluation covers pick and place in one environment on a few objects, so per-object numbers are noisy. Policies struggle with precise grasps and often grip too loosely, letting a bottle slip; the state representation was chosen from preliminary tests, not a full ablation.

Videos

R1 Pro picks a bag of dried mango from the shelf into the basket.

R1 Pro picks a bag of peas from the shelf into the basket.

Figures

From Figure 1: (b) Operators collect demonstrations with battery-powered handheld grippers whose soft, angled fingers... — From Figure 1: (b) Operators collect demonstrations with battery-powered handheld grippers whose soft, angled fingers match those of the robot, so the system can be used anywhere.

From Figure 1: (a) The handheld UMI gripper with soft fingertips, a wrist camera and a tracked Meta Quest controller.

From Figure 1: (d) Because demonstrations record end-effector poses, policies deploy on any robot with matching gripp... — From Figure 1: (d) Because demonstrations record end-effector poses, policies deploy on any robot with matching grippers that can reach those poses, here the Galaxea R1 Pro.

From Figure 2: Success rate on each held-out object, grouped by whether its shape is similar to or different from the... — From Figure 2: Success rate on each held-out object, grouped by whether its shape is similar to or different from the mid-training objects. Left: R1 Pro, mid-training and fine-tuning on UMI data. Right: R1 Lite, the same policies fine-tuned on teleoperation data (hatched). Error bars show 95% Wilson intervals.

Abstract

Training robot policies for multi-task pick and place requires demonstrations on many objects, which handheld Universal Manipulation Interfaces (UMIs) can collect quickly and cheaply. However, existing UMIs depend on external base stations, use rigid fingers that give no signal about contact forces, or use parallel-jaw fingers that limit grip strength. We present a battery-powered handheld interface, built on YUBI, whose soft, angled fingers wrap around objects, deform visibly under load as a visual proxy for force, and match the grippers of the robot. With it, we collect a 99.7-hour dataset over 53 objects, train $π_{0.5}$ policies on 49 of them, and hold out the other 4 as unseen objects. We find that a top camera raises success by 13 points even when only about half of the data includes it, and that mid-training on our multi-task data raises success after fine-tuning on unseen objects from 35% to 57%. Finger design, dataset, and data processing code: https://lute-corp.github.io/soft-yubi-site/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong