Reactive Human-to-Robot Handovers of Arbitrary Objects
TL;DR A vision-based system lets a Franka arm take unknown objects from a person's hand by segmenting the hand, generating temporally consistent 6-DOF grasps on the remaining points and picking among them with a reactive planner. In a six-person study it took all 10 household test objects from every participant, with an 81.8% average success rate and 1.3 attempts per object.
What they do
Prior human-to-robot handover systems were tested on a handful of objects, or needed the object to belong to a detector’s known categories. This paper builds a vision-only system that takes unknown rigid or deformable objects from a person’s hand and reacts while the person moves. It is tested on 26 household objects drawn from a list of items people with ALS most need fetched.
How it works
- Segment the hand, not the object. Azure Kinect body tracking locates the palm and the point cloud within 20 cm is cropped. A ResNet-50 FPN network segments the hand in RGB; every other point in the crop is labeled object. Training masks come from cropped bare-hand point clouds, with no manual annotation.
- Temporally consistent grasps. 6-DOF GraspNet samples grasps on the object points. Each frame, the previous grasps are shifted by up to 2 cm and accepted by Metropolis-Hastings on the GraspNet evaluator score; grasps colliding with the hand are removed, and new ones are sampled when too few remain.
- Reactive grasp selection. A four-state task model (wait, approach, grasp, place) is re-checked continuously. Grasps and their 180° flipped copies are scored by quality and by distance to the previous choice and the home pose. The first with an IK solution and collision-free paths sets a 10 cm standoff target, updated at about 10 Hz and followed with Riemannian Motion Policies.
Results
- Banana, bottle and cereal box held flat, at 45° and at 90°: 90% success overall, 11.4 s mean handover time, every object taken within three attempts.
- Object rotated after the robot starts moving: 100% success for 0-45° rotations, 69% for 0-90°.
- User study (6 participants, 10 objects): every object was taken from every participant, with 10.7 s mean approach time, 1.3 attempts and 81.8% success. Mug, pen and spoon were hardest (54.5%, 50%, 66.7%). Questionnaire ratings were around 4 or above.
Limitations
Failures came from missing depth on dark surfaces, objects segmented as hand, and nearby objects adding noise to the cropped point cloud. Participants found the robot slow and not always sharp in following their motion; the authors name inverse kinematics cost as a bottleneck. The six participants were recruited from the lab, and performance was measured only for the 10 Household-A objects.
Figures
Abstract
Human-robot object handovers have been an actively studied area of robotics over the past decade; however, very few techniques and systems have addressed the challenge of handing over diverse objects with arbitrary appearance, size, shape, and rigidity. In this paper, we present a vision-based system that enables reactive human-to-robot handovers of unknown objects. Our approach combines closed-loop motion planning with real-time, temporally-consistent grasp generation to ensure reactivity and motion smoothness. Our system is robust to different object positions and orientations, and can grasp both rigid and non-rigid objects. We demonstrate the generalizability, usability, and robustness of our approach on a novel benchmark set of 26 diverse household objects, a user study with naive users (N=6) handing over a subset of 15 objects, and a systematic evaluation examining different ways of handing objects. More results and videos can be found at https://sites.google.com/nvidia.com/handovers-of-arbitrary-objects.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.