Multimodal Fusion of EMG and Vision for Human Grasp Intent Inference in Prosthetic Hand Control

Mehrshad Zandigohar, Mo Han, Mohammadreza Sharif, Sezen Yagmur Gunay, Mariusz P. Furmanek, Mathew Yarossi, Paolo Bonato, Cagdas Onal, Taskin Padir, Deniz Erdogmus, Gunar Schirner

Seminal April 2021

TL;DR Predict which of 13 grasp types a prosthetic-hand user intends by multiplying the probabilities of a forearm-EMG classifier and a gaze-guided camera detector. During reaching, fusion is 95.3% accurate, against 81.64% for EMG and 80.5% for vision alone.

What they do

EMG-controlled prosthetic hands suffer from electrode shift, fatigue and impedance changes, while camera-based grasp classification suffers from occlusion and lighting. This paper fuses the two to infer the intended grasp type. Forearm EMG and the eye tracker’s world camera plus gaze each give a probability over 13 grasp types, and Bayesian fusion combines them. Accuracy is analysed over time across the reach, grasp, return and rest phases, offline, on 5 healthy subjects.

How it works

Dataset. 12 EMG channels at 1562.5 Hz and a Pupil Core eye tracker (60 Hz world camera). Subjects moved 54 objects around a 3x2 grid of positions, 6 trials per object, in clockwise and counterclockwise sessions; 14 labels (13 grasps plus open palm/rest).
EMG. Greedy Gaussian Segmentation splits each trial into reach, grasp, return and rest without supervision; rest is labelled open palm. RMS, MAV and variance features on 320 ms windows feed an extra-trees classifier (50 trees).
Vision. YOLOv4 is fine-tuned to output grasp-type probabilities per detected object, and the box closest to the gaze point is used. Copy-paste augmentation puts objects onto NYU Depth V2 indoor backgrounds, using Mask R-CNN masks refined with GrabCut and Poisson blending.
Fusion. A graphical model V → L → M with a uniform prior reduces to picking the grasp that maximises P(L M)P(L V), computed independently at each time step.

Results

Reaching phase: EMG 81.64%, vision 80.5%, fusion 95.3%. Fusion beats both in every phase (89.7-94.89%), with 92.93% overall against 41.85% for EMG and 81.46% for vision.
The modalities complement each other: during grasping vision drops to 66.22% (occlusion) while EMG holds 78.66%; at rest EMG is at chance (6.33%) while vision reaches 90.69%.
Smoothing the fused decisions over time raises average accuracy to 96.8%.
The grasp detector reaches 84.97% test mAP; per-subject EMG gesture accuracy ranges from 81.8% to 90.6%.

Limitations

Five healthy subjects, and all analysis is offline with no prosthetic hand actuated. Four of the 12 muscles are intrinsic hand muscles that most amputees lack, and amputees would need subject-specific training. The fusion is memoryless.

Figures

From Figure 1: System overview. A visual classifier on the eye-tracker camera and gaze and an EMG classifier each out... — From Figure 1: System overview. A visual classifier on the eye-tracker camera and gaze and an EMG classifier each output grasp-type probabilities, which are fused to choose the prosthetic hand's grasp.

From Figure 2: Data collection setup, with an eye tracker carrying a world camera and EMG electrodes on the forearm w... — From Figure 2: Data collection setup, with an eye tracker carrying a world camera and EMG electrodes on the forearm while the subject grasps an object.

From Figure 5: Background generalization. Mask R-CNN masks are refined and the object is copy-pasted and blended onto... — From Figure 5: Background generalization. Mask R-CNN masks are refined and the object is copy-pasted and blended onto new indoor backgrounds.

From Figure 10: Average validation accuracy over time for EMG, vision and fusion. The two sources complement each oth... — From Figure 10: Average validation accuracy over time for EMG, vision and fusion. The two sources complement each other, and fusion is the most accurate throughout.

Abstract

Objective: For transradial amputees, robotic prosthetic hands promise to regain the capability to perform daily living activities. Current control methods based on physiological signals such as electromyography (EMG) are prone to yielding poor inference outcomes due to motion artifacts, muscle fatigue, and many more. Vision sensors are a major source of information about the environment state and can play a vital role in inferring feasible and intended gestures. However, visual evidence is also susceptible to its own artifacts, most often due to object occlusion, lighting changes, etc. Multimodal evidence fusion using physiological and vision sensor measurements is a natural approach due to the complementary strengths of these modalities. Methods: In this paper, we present a Bayesian evidence fusion framework for grasp intent inference using eye-view video, eye-gaze, and EMG from the forearm processed by neural network models. We analyze individual and fused performance as a function of time as the hand approaches the object to grasp it. For this purpose, we have also developed novel data processing and augmentation techniques to train neural network components. Results: Our results indicate that, on average, fusion improves the instantaneous upcoming grasp type classification accuracy while in the reaching phase by 13.66% and 14.8%, relative to EMG (81.64% non-fused) and visual evidence (80.5% non-fused) individually, resulting in an overall fusion accuracy of 95.3%. Conclusion: Our experimental data analyses demonstrate that EMG and visual evidence show complementary strengths, and as a consequence, fusion of multimodal evidence can outperform each individual evidence modality at any given time.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong