Open-TeleVision: Teleoperation with Immersive Active Visual Feedback

Xuxin Cheng, Jialong Li, Shiqi Yang, Ge Yang, Xiaolong Wang

Seminal July 2024

An operator at MIT teleoperates the H1 at UC San Diego in real time.

TL;DR A VR teleoperation system that streams stereo video from an actuated camera on a humanoid's head to the operator's headset, while the robot's head, arms and hands follow the operator's. ACT policies trained on 10-20 of its demonstrations reach 87-100% stage success on four long-horizon tasks on a Unitree H1.

What they do

In most teleoperation setups the operator watches the robot directly or through a fixed camera: they must be in the same room, the robot’s own body occludes the view, and close-up work is hard. Open-TeleVision mounts a stereo camera on an actuated neck and streams its view to a VR headset. The camera turns with the operator’s head, the arms and hands mirror theirs, and the operator can be across the country. Policies trained on the demonstrations also move the head. Tested on a Unitree H1 with 6-DoF Inspire hands and a Fourier GR-1 with grippers.

How it works

Loop. An Apple Vision Pro sends hand, head and wrist poses to a web server (built on Vuer); a ZED Mini streams 480x640 stereo video per eye back. The loop runs at 60 Hz.
Arms. Wrist positions relative to the head set end-effector positions; orientations follow the absolute wrist orientations. Closed-loop IK in Pinocchio, with a nullspace joint offset near manipulability limits.
Hands. dex-retargeting with seven fingertip vectors for dexterous hands and one thumb-index vector for grippers.
Policy. ACT with a DinoV2 backbone on both stereo images, predicting absolute joint positions including the neck.

Results

H1 stage success: can sorting 92% pick / 88% place, insertion 90% / 87%, folding and unloading 100% on every stage. Left image only drops sorting picks to 46% and insertions to 63%; a ResNet18 backbone also does worse.
The cropped active view trains 2x faster than a wide-angle static view and runs at 83 Hz instead of 42 Hz.
User study (4 participants): stereo gives 100% success on every task, against 71% (insertion) and 50% (unloading) with mono, and shorter mean times on all four (58 s vs 85 s for insertion).

Limitations

GR-1 can sorting places correctly only 60% of the time, which the authors attribute to the gripper hiding the can’s colour and ACT’s one-second action chunk; labelled cans raise it to 100%. There is no haptic feedback and no way to relabel expert data, and only the neck, arms and end-effectors are used; a mobile version is left to future work.

Videos

Can Insertion: the H1 places cans into the container slots one by one, 8x speed.

Unloading: the H1 extracts tubes from a sorting box one at a time, 8x speed.

Figures

From Figure 1: Autonomous runs (top: unloading with in-hand passing, H1 and GR-1 can sorting, can insertion, towel fo... — From Figure 1: Autonomous runs (top: unloading with in-hand passing, H1 and GR-1 can sorting, can insertion, towel folding) and teleoperated tasks (bottom: earplug packing, drilling, pipetting, two operators driving two robots with the H1 operator in Boston and the robots in San Diego, and interaction with people).

From Figure 2: The VR headset streams hand, head and wrist poses to a server, which retargets them to robot joint tar... — From Figure 2: The VR headset streams hand, head and wrist poses to a server, which retargets them to robot joint targets while the robot streams stereo video back (left). The collected demonstrations train an action chunking transformer per task (right).

From Figure 7: A static wide-angle view (1280x720, 102 x 57 degree field of view) versus 640x480 crops like the activ... — From Figure 7: A static wide-angle view (1280x720, 102 x 57 degree field of view) versus 640x480 crops like the active camera sees, for H1 Can Sorting (left) and Unloading (right). Red marks are the points of interest each task needs to see.

From Figure 6: Cans placed on a 4x4 grid of 3 cm cells (left) and the number of successful H1 picks out of 5 trials a... — From Figure 6: Cans placed on a 4x4 grid of 3 cm cells (left) and the number of successful H1 picks out of 5 trials at each location (right).

Abstract

Teleoperation serves as a powerful method for collecting on-robot data essential for robot learning from demonstrations. The intuitiveness and ease of use of the teleoperation system are crucial for ensuring high-quality, diverse, and scalable data. To achieve this, we propose an immersive teleoperation system Open-TeleVision that allows operators to actively perceive the robot's surroundings in a stereoscopic manner. Additionally, the system mirrors the operator's arm and hand movements on the robot, creating an immersive experience as if the operator's mind is transmitted to a robot embodiment. We validate the effectiveness of our system by collecting data and training imitation learning policies on four long-horizon, precise tasks (Can Sorting, Can Insertion, Folding, and Unloading) for 2 different humanoid robots and deploy them in the real world. The system is open-sourced at: https://robot-tv.github.io/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong