emg2qwerty: A Large Dataset with Baselines for Touch Typing using Surface Electromyography
TL;DR A public dataset of wrist sEMG recorded while 108 people touch type, 346 hours in 1,135 sessions, with keylogger ground truth. A speech-recognition-style baseline decodes keystrokes at 6.95% character error rate once personalized, but over 50% on unseen users.
What they do
Wrist-worn surface EMG could be a private, always-available text input, but there has been no large public dataset with a well-defined task. emg2qwerty records sEMG at both wrists while people touch type prompted text on a QWERTY keyboard, with a keylogger providing ground truth. The task is to predict the key-presses from sEMG alone, and the benchmark measures generalization to unseen users and how well a model personalizes to them.
How it works
- Hardware. One sEMG-RD wristband per wrist, each with 16 dry differential electrode pairs sampled at 2 kHz, streaming over Bluetooth with no skin preparation.
- Data. 108 screened touch typists, 1,135 sessions, 346 hours and 5.26 million keystrokes. Bands are taken off and put back on between sessions, so electrode placement varies.
- Benchmark. 100 users train a generic model. 8 held-out users each keep 2 sessions for validation and 2 for testing; the rest train personalized models. The metric is character error rate (CER).
- Baseline. Borrowed from speech recognition: log spectrogram features, SpecAugment plus electrode-rotation and left-right timing jitter augmentation, a Time Depth Separable ConvNet with a 1 s receptive field, CTC loss, and a 6-gram character language model with a backspace-aware beam-search decoder.
Results
- The generic model is unusable on unseen users: 55.38% test CER, 51.78% with the language model.
- Personalized models trained from scratch reach 15.38% test CER, 9.55% with the language model.
- Finetuning the generic model on each user is best: 11.28% test CER, 6.95% with the language model, and 3.16% for the best user. The authors note models become usable at about 10% CER or less.
Limitations
Models require touch typing on a QWERTY keyboard and cover only English. Data was collected on a physical keyboard at a desk, so typing forces and sEMG amplitude may differ from keyboard-free use. The research-grade wristband is proprietary, which rules out human-in-the-loop testing by others. Deployment would need the model to run on the wristband within its compute and battery limits.
Figures
Abstract
Surface electromyography (sEMG) non-invasively measures signals generated by muscle activity with sufficient sensitivity to detect individual spinal neurons and richness to identify dozens of gestures and their nuances. Wearable wrist-based sEMG sensors have the potential to offer low friction, subtle, information rich, always available human-computer inputs. To this end, we introduce emg2qwerty, a large-scale dataset of non-invasive electromyographic signals recorded at the wrists while touch typing on a QWERTY keyboard, together with ground-truth annotations and reproducible baselines. With 1,135 sessions spanning 108 users and 346 hours of recording, this is the largest such public dataset to date. These data demonstrate non-trivial, but well defined hierarchical relationships both in terms of the generative process, from neurons to muscles and muscle combinations, as well as in terms of domain shift across users and user sessions. Applying standard modeling techniques from the closely related field of Automatic Speech Recognition (ASR), we show strong baseline performance on predicting key-presses using sEMG signals alone. We believe the richness of this task and dataset will facilitate progress in several problems of interest to both the machine learning and neuroscientific communities. Dataset and code can be accessed at https://github.com/facebookresearch/emg2qwerty.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.