AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System
TL;DR One camera-based teleoperation system that drives different robot arms and dexterous hands, in simulation or the real world, needing only each robot's URDF. On the same xArm6 and Allegro hardware as a robot-specific prior system, it beats that system on 8 of its 10 real-world tasks and ties on the other 2.
What they do
Vision-based teleoperation of dexterous arm-hand robots is usually built around one robot, one camera setup and one simulator or real rig. AnyTeleop is a single system that tracks the operator’s hand with ordinary cameras and drives different arms and hands in IsaacGym, SAPIEN or the real world. It adapts to a new robot from its URDF (non-humanlike hands such as a D’Claw also need a manual finger mapping). A browser-based viewer supports remote operation and several operators sharing one scene.
How it works
- Hand detection. MediaPipe gives 21 finger keypoints from RGB. Wrist pose comes from depth plus PnP, or, with RGB only, from a FrankMocap-style network that predicts hand scale. No extrinsic camera calibration is needed.
- Multi-camera fusion. Cameras are calibrated against each other using the hand over the first 50 frames. At each step, the camera whose predicted SMPL-X hand shape deviates least from the reference is trusted.
- Retargeting and motion. An optimization matches human and robot keypoint vectors within joint limits, with a smoothness penalty. CuRobo turns 25 Hz wrist targets into collision-free 120 Hz arm trajectories.
- Deployment. Modules share a common interface, communicate over the network and ship as a Docker image.
Results
- Real robot, on the ten Robotic Telekinesis tasks with the same xArm6 and Allegro hand: higher success on 8 tasks and equal on 2, e.g. 0.7 vs 0.3 on two-cup stacking. Most gains are on thin-walled cups.
- Imitation learning (DAPG, 50 demos per task) in simulation: demos from AnyTeleop beat those from Qin et al.’s system on 5 of 6 tasks, e.g. arm-hand relocate 70.0 vs 40.3 and open door 79.7 vs 51.3. Floating-hand flip mug is the exception (47.3 vs 51.3).
- Play Piano in IsaacGym: one RGB camera takes 109 s with 28.1% wrong keys; two RGB-D cameras take 74 s with 12.5%.
Limitations
Fast hand motion loses tracking and forces a pause and re-detection. Hand pose is unreliable under self-occlusion, which extra cameras mitigate. Running every module on one computer struggles to reach the 120 Hz motion rate, so control runs on a separate machine. Collaborative manipulation is shown only as a simulated human-to-robot handover.
Videos
Figures
Abstract
Vision-based teleoperation offers the possibility to endow robots with human-level intelligence to physically interact with the environment, while only requiring low-cost camera sensors. However, current vision-based teleoperation systems are designed and engineered towards a particular robot model and deploy environment, which scales poorly as the pool of the robot models expands and the variety of the operating environment increases. In this paper, we propose AnyTeleop, a unified and general teleoperation system to support multiple different arms, hands, realities, and camera configurations within a single system. Although being designed to provide great flexibility to the choice of simulators and real hardware, our system can still achieve great performance. For real-world experiments, AnyTeleop can outperform a previous system that was designed for a specific robot hardware with a higher success rate, using the same robot. For teleoperation in simulation, AnyTeleop leads to better imitation learning performance, compared with a previous system that is particularly designed for that simulator. Project page: https://yzqin.github.io/anyteleop/.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.