Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

RUMI: A Robotiq-Matched Universal Manipulation Interface for Robot Learning with Augmented Reality and Haptic Feedback

Kevin Avdiu, Seongjin Bien, Yannik Blei, Philine Meister, Johannes Hechtl, Philipp Schmitt, Kai Wurm, Georg von Wichert, Wolfram Burgard

November 2026

TL;DR A handheld gripper that copies the Robotiq 2F-85's kinematics, takes swappable force or tactile fingertips, shows the target robot in AR while recording and locks its linkage for haptic feedback during teleoperation. With AR, novices' demonstrations hit joint limits in 3/50 episodes instead of 16/50, and fingertip force sensing lifts real-robot gear-assembly success from 7/20 to 20/20.

What they do

RUMI is a handheld data-collection gripper that copies the Robotiq 2F-85’s nonlinear finger kinematics, finger mount and wrist-camera pose, so demonstrations match a common industrial gripper. The same device serves robot-free collection, with an AR view of the target robot in a Quest 3 headset, and teleoperation of simulated or real robots with haptic feedback from a mechanical blocker.

How it works

  • Hardware. A scissor linkage derived from the 2F-85 CAD model couples thumb and index finger to the fingers; an encoder reads the aperture and an index-pad CoinFT sensor measures the operator’s squeeze. Fingertips swap between stock Robotiq, CoinFT force/torque and DIGIT tactile. A D405 sits at the robot’s wrist-camera pose; an attached Quest 3 controller gives the 6-DoF pose.
  • AR digital shadow. The IRIS app renders a MuJoCo model of the target robot, registered with QR codes, and solves IK for the tracked pose so the operator sees infeasible poses while recording. A catch-to-teleop step aligns each start pose with the projected end effector.
  • Haptic teleoperation. Before contact the linkage moves freely and sets the aperture. Once the remote gripper touches something, a micro servo locks the linkage and further squeeze becomes a grasp-force command.

Results

  • User study (5 novices, 50 box pick-and-place episodes per condition): joint-limit violations in 3/50 episodes with AR against 16/50 without, joint-velocity violations in 11/50 against 29/50.
  • ACT policies on a Franka FR3, 20 trials each, with against without fingertip sensing: gear assembly 20/20 vs 7/20 and whiteboard wiping 16/20 vs 7/20 (CoinFT); screw tightening 9/20 vs 3/20 and box sorting 17/20 vs 9/20 (DIGIT).
  • Simulated packing with the blocker: demonstrators used 5.66 N on a fragile bulb and 22.82 N on a heavy block, and policies succeed 94% and 97%. Without it, bulb demonstrations average 18.46 N and every bulb rollout breaks the 10 N limit; the block still succeeds 99%.

Limitations

Observations come from the wrist camera only. Beyond the binary blocker cue and controller vibration there is no feedback of external forces. Handheld jitter makes the screw-tightening policy slip. The haptic evaluation is in simulation only.

Figures

From Figure 2: The hardware (Robotiq 2F-85-matched kinematics, D405 wrist camera, rotary encoder, Quest controller, f...
From Figure 2: The hardware (Robotiq 2F-85-matched kinematics, D405 wrist camera, rotary encoder, Quest controller, fingertip force/torque sensing), how the hand holds it, the nonlinear transmission with its blocker, and which contact and force paths are active in the VR, teleoperation and MR modes.
From Figure 4: In-the-wild data collection, with the target robot shown as an AR digital shadow.
From Figure 4: In-the-wild data collection, with the target robot shown as an AR digital shadow.
From Figure 3: Whiteboard wiping, one of the two tasks recorded with CoinFT fingertips: pick up the eraser and wipe o...
From Figure 3: Whiteboard wiping, one of the two tasks recorded with CoinFT fingertips: pick up the eraser and wipe off the star.
From Figure 5: Initial poses of all user-study episodes with and without AR, in translation (left) and rotation (right).
From Figure 5: Initial poses of all user-study episodes with and without AR, in translation (left) and rotation (right).
Abstract

Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale, force-aware datasets remain scarce. Handheld interfaces can help by transmitting interaction forces directly through the operator's arm, but most existing designs are tailored to simple linear grippers and do not match the nonlinear mechanics of widely used robotic grippers. We present RUMI, a handheld universal manipulation interface matched to the Robotiq 2F-85 with exchangeable fingertip sensors that preserves the target gripper geometry while enabling natural input before contact and force measurement after contact. By integrating the tracking system into a virtual reality headset, our method provides real-time augmented reality feedback about the kinematic feasibility of motion on the target robot platform. We further introduce a haptic feedback mechanism that supports force-aware teleoperation for simulated and real robots. A user study shows fewer episodes containing joint-limit violations with live robot visualization and pose initialization (3/50) than without augmented reality (16/50). We additionally train imitation policies from demonstrations, collected using two different fingertip modalities enabled by the gripper-matched hardware, and assess the haptic interface's capability for collecting force-aware teleoperation datasets in simulation.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.