Gaze-based, Context-aware Robotic System for Assisted Reaching and Grasping
TL;DR The user looks at an object and a UR10 arm plus a robotic glove reach for it, grasp it and use it, sequenced by a finite state machine built on action grammars. 3D gaze is estimated to within 4.68 cm, and with 5 users every reach succeeded, pick-and-place succeeded 96% of the time and pick-pour-place 76%.
What they do
People with upper-limb impairments usually keep control of their eyes. This system lets a user drive reach-and-grasp assistance with gaze alone: they look at an object, and a UR10 arm attached to their wrist reaches for it while a Bioservo Carbonhand glove closes their hand. Multi-step tasks such as pick, pour and place are chained by a finite state machine that encodes which actions make sense on which objects (“action grammars”).
How it works
- 3D gaze. An Intel RealSense D435 is mounted on SMI eye-tracking glasses. The 2D gaze pixel and its depth give a 3D gaze point, which OptiTrack head tracking transforms into the robot frame.
- Context and intent. A CNN detects objects in the egocentric image. Gazing at an object’s right-hand side for 15 gaze samples signals intent to act, so users can look around freely without moving the robot.
- Action grammar FSM. Objects are coded by graspability and pourability; states by grip and held object. Fruit can be placed on the table or in the bowl, a cup can also be poured into the bowl. Glove tendon and fingertip force data detect grasp failures, which release the grip and reset.
Results
- 3D gaze accuracy (8 subjects, robot-held target at 10 random points): mean Euclidean error 4.68 cm, with no significant difference between subjects. Error correlated with target height, which the authors attribute to poorer pupil tracking at extreme vertical gaze.
- Full system (5 subjects, 5 repeats per task): reaching succeeded 100% of the time, pick-and-place 96% and pick-pour-place 76%. Training took under 5 minutes per participant.
- Most failures came in pouring: the cup shifted in the grasp, so it dropped or was set down not upright.
Limitations
The magnetic wrist attachment does not support pouring well. Users get no feedback before the robot acts, and some rated it unpredictable on the usability questionnaire. Only 5 healthy, young, male participants tested the full system. Head tracking needs an external OptiTrack setup.
Figures
Abstract
Assistive robotic systems endeavour to support those with movement disabilities, enabling them to move again and regain functionality. Main issue with these systems is the complexity of their low-level control, and how to translate this to simpler, higher level commands that are easy and intuitive for a human user to interact with. We have created a multi-modal system, consisting of different sensing, decision making and actuating modalities, leading to intuitive, human-in-the-loop assistive robotics. The system takes its cue from the user's gaze, to decode their intentions and implement low-level motion actions to achieve high-level tasks. This results in the user simply having to look at the objects of interest, for the robotic system to assist them in reaching for those objects, grasping them, and using them to interact with other objects. We present our method for 3D gaze estimation, and grammars-based implementation of sequences of action with the robotic system. The 3D gaze estimation is evaluated with 8 subjects, showing an overall accuracy of $4.68\pm0.14cm$. The full system is tested with 5 subjects, showing successful implementation of $100\%$ of reach to gaze point actions and full implementation of pick and place tasks in 96\%, and pick and pour tasks in $76\%$ of cases. Finally we present a discussion on our results and what future work is needed to improve the system.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.