Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

HandoverSim: A Simulation Framework and Benchmark for Human-to-Robot Object Handovers

Yu-Wei Chao, Chris Paxton, Yu Xiang, Wei Yang, Balakumar Sundaralingam, Tao Chen, Adithyavairavan Murali, Maya Cakmak, Dieter Fox

Seminal May 2022

A real DexYCB motion capture, then the same grasp-and-offer replayed in simulation.

TL;DR A PyBullet benchmark for human-to-robot handovers, where a simulated hand replays 1,000 real motion captures and a Panda arm has to take the object without touching the hand. The best of four existing baselines succeeds 64.58% of the time, and the baselines rank the same way they did in a real user study.

What they do

Testing human-to-robot handovers usually needs a person in the loop, and every study picks its own objects and metrics. HandoverSim is a simulation benchmark where a Franka Panda takes objects from a simulated human hand that follows real motion capture. The authors call it the first simulation benchmark for human-to-robot handovers.

How it works

  • Giver. Each of DexYCB’s 1,000 captures (10 subjects, 20 YCB objects, 5 trials each) becomes a PyBullet scene. The MANO hand and the object follow their captured trajectories directly. The object has its own actuators and hand-object collisions are switched off. At the end of the capture, the hand holds still and waits.
  • Release. The object is let go once both gripper fingers have touched it for 0.1 s. It also drops if it touches only other parts of the robot for 0.1 s. After that, only gravity and the robot act on it.
  • Task. A Gym environment whose actions are 9-D joint targets for a PD controller. Success: the gripper holds the object inside a goal sphere for 0.1 s. Failure: the robot touches the hand, the object drops, or 13 s pass.
  • Splits. Scenes with two objects the gripper cannot handle are dropped. The remaining 900 are split four ways: default, unseen subjects, unseen handedness and unseen objects.

Results

  • Baselines get ground-truth states and no training on the benchmark. On the default split, OMG Planner succeeds 62.50% of the time, but 27.78% of runs end in hand contact and it takes 9.72 s on average. Yang et al. succeeds 64.58% (17.36% contact, 4.90 s). GA-DDPG succeeds 50.00% when it waits for the hand to stop and 36.81% when it does not.
  • In a real 6-person study, Yang et al. beat GA-DDPG (82% vs 80% success). It does so in simulation too (62.78% vs 55.00% on unseen subjects), and it is faster in both.
  • Success is higher in the real study (GA-DDPG: 80% vs 55.00%). The authors attribute this to cooperative users adjusting the object.

Limitations

The simulated human never reacts to the robot. Hand motion during and after the release is not simulated because no data exists for it. The grasp is not physically simulated: the object moves on its own actuators. Baselines see perfect states, so the reported times include no perception latency.

Videos

Many of the benchmark's 1,000 simulated handover scenes playing side by side.
Full video (YouTube)

Figures

From Figure 1: the simulation environment. The green human hand offers an object to the Panda arm; the red sphere is ...
From Figure 1: the simulation environment. The green human hand offers an object to the Panda arm; the red sphere is the goal region.
From Figure 2: a simulated left hand lifting a can from the table, replaying its captured motion.
From Figure 2: a simulated left hand lifting a can from the table, replaying its captured motion.
From Figure 4: a successful handover by the OMG Planner baseline, with the gripper taking a cracker box from the hand.
From Figure 4: a successful handover by the OMG Planner baseline, with the gripper taking a cracker box from the hand.
Abstract

We introduce a new simulation benchmark "HandoverSim" for human-to-robot object handovers. To simulate the giver's motion, we leverage a recent motion capture dataset of hand grasping of objects. We create training and evaluation environments for the receiver with standardized protocols and metrics. We analyze the performance of a set of baselines and show a correlation with a real-world evaluation. Code is open sourced at https://handover-sim.github.io.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.