Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning

Runyu Ding, Yuzhe Qin, Jiyue Zhu, Chengzhe Jia, Shiqi Yang, Ruihan Yang, Xiaojuan Qi, Xiaolong Wang

Seminal July 2024

Autonomous bimanual policy sweeping trash with a broom and dustpan, 4x speed.

TL;DR A real-time system that teleoperates two arms with dexterous hands from Apple Vision Pro hand tracking, with built-in collision and singularity avoidance and $1.2 vibration motors for touch feedback. Policies trained on its demonstrations average a 22% higher success rate than policies trained on AnyTeleop+ data.

What they do

Bunny-VisionPro teleoperates a 24-DoF bimanual robot, two UFactory xArm-7 arms with 6-DoF Ability hands, from an Apple Vision Pro. The operator’s finger poses drive the hands, their wrist poses drive the arms, and touch on the robot’s fingertips comes back as vibration on the operator’s fingers. The aim is to collect good demonstrations for bimanual and long-horizon imitation learning.

How it works

Hand retargeting. SQP matches scaled human fingertip keypoint vectors to the robot’s, with a smoothness penalty. The passive joints of the hand’s four-bar linkages are computed from the active joints, and their gradients are added back to the active joints instead of being handled as equality constraints. This makes it 10.2x faster, and it runs at 300 Hz on one CPU core.
Arm control. One objective combines IK to the wrist pose, a manipulability penalty that switches on only near singularity, and self-collision costs between spheres that approximate each link. It runs above 60 Hz without a GPU.
Haptics. The hand’s FSR tactile readings are zero-drift calibrated and low-pass filtered. They then drive $1.2 ERM vibration motors in finger cots through PWM.

Results

Telekinesis benchmark: matches or beats prior systems on 9 of 10 tasks. Scissor pickup is worse because of the hand’s limited DoF.
Six custom tasks against AnyTeleop+ (AnyTeleop with Vision Pro tracking on the same robot): 11% higher success and 45% less completion time, while AnyTeleop+ moves the arm joints 43% more. Collecting 30 sweep-floor demos took 45.4 min instead of 129.4.
Imitation learning with ACT, Diffusion Policy and DP3: 22% higher average success than with AnyTeleop+ data, plus 14% higher under new object poses and 26% higher with unseen objects. Long-horizon tasks reach 73% sub-task and 38% full-task success from 30 demos.
Haptics study (5 untrained operators): success held or improved in 9 of 10 comparisons. Adding tactile input to policies did not consistently help.

Limitations

Vision Pro hand tracking is inaccurate when fingers occlude each other, which makes commands jerky. Light touches cause vibration changes too small to feel. The haptic device was built after data collection, so neither the teleoperation comparison nor the imitation-learning demos used it.

Videos

Teleoperated ball moving, one of the two haptics user-study tasks.

Haptics demo, touching the robot fingertips beside the vibration-motor finger cots.

Figures

From Figure 1: (a) Hand poses tracked by Apple Vision Pro become robot motion commands, and the robot sends vision an... — From Figure 1: (a) Hand poses tracked by Apple Vision Pro become robot motion commands, and the robot sends vision and touch back through the headset and actuator-equipped finger cots. (b) The short-horizon (left) and long-horizon (right) tasks used for teleoperation and imitation learning.

From Figure 2: Finger poses control the robot hands and wrist poses control the arms; the operator gets visual and ha... — From Figure 2: Finger poses control the robot hands and wrist poses control the arms; the operator gets visual and haptic feedback, and the system runs loop-joint retargeting, singularity avoidance, collision avoidance and haptic feedback in real time.

From Figure 4: Success rate and completion time of five untrained operators (A-E) on toy handover and ball moving, wi... — From Figure 4: Success rate and completion time of five untrained operators (A-E) on toy handover and ball moving, with and without haptic feedback.

Abstract

Teleoperation is a crucial tool for collecting human demonstrations, but controlling robots with bimanual dexterous hands remains a challenge. Existing teleoperation systems struggle to handle the complexity of coordinating two hands for intricate manipulations. We introduce Bunny-VisionPro, a real-time bimanual dexterous teleoperation system that leverages a VR headset. Unlike previous vision-based teleoperation systems, we design novel low-cost devices to provide haptic feedback to the operator, enhancing immersion. Our system prioritizes safety by incorporating collision and singularity avoidance while maintaining real-time performance through innovative designs. Bunny-VisionPro outperforms prior systems on a standard task suite, achieving higher success rates and reduced task completion times. Moreover, the high-quality teleoperation demonstrations improve downstream imitation learning performance, leading to better generalizability. Notably, Bunny-VisionPro enables imitation learning with challenging multi-stage, long-horizon dexterous manipulation tasks, which have rarely been addressed in previous work. Our system's ability to handle bimanual manipulations while prioritizing safety and real-time performance makes it a powerful tool for advancing dexterous manipulation and imitation learning.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong