NOIR: Neural Signal Operated Intelligent Robots for Everyday Activities

Ruohan Zhang, Sharon Lee, Minjune Hwang, Ayano Hiranaka, Chen Wang, Wensi Ai, Jin Jie Ryan Tan, Shreya Gupta, Yilun Hao, Gabrael Levine, Ruohan Gao, Anthony Norcia, Li Fei-Fei, Jiajun Wu

★ Must read November 2023

The Franka arm cooks sukiyaki under EEG control (8x speed, decoding wait periods omitted).

TL;DR NOIR lets a person command a robot arm or mobile manipulator through non-invasive EEG by decoding what object to act on, how, and where, then running a parameterized primitive skill. Three users completed 20 long-horizon household tasks in 1.83 attempts on average.

What they do

NOIR is a brain-robot interface built on non-invasive, saline-based EEG. It follows hierarchical shared autonomy: the person watches the scene on a screen and communicates a high-level goal through brain signals, and the robot carries it out with a library of parameterized primitive skills such as Pick(obj-A) or MoveTo(x,y). The system is tested on 20 everyday activities chosen from the BEHAVIOR benchmark and Activities of Daily Living, on a Franka Emika Panda arm (16 tabletop tasks) and a PAL Tiago (4 mobile manipulation tasks).

How it works

What. OWL-ViT segments the objects and each mask flickers on screen at its own frequency (6, 7.5, 8.57 or 10 Hz). Looking at one for 10 s evokes an SSVEP, which Canonical Correlation Analysis classifies.
How and where. The user picks a skill by motor imagery (left hand, right hand, legs or rest), decoded from 5 s of EEG with common spatial patterns plus quadratic discriminant analysis after 10 minutes of calibration. A 2-way classifier then moves a cursor along each axis to set the skill parameter.
Safety. A jaw clench, detected from muscle tension in 500 ms windows, confirms or rejects each selection.
Robot learning. A retrieval model on R3M features trained with a triplet loss proposes the next object-skill pair, and one-shot DINOv2 feature matching proposes the parameter location, so the user has less to decode.

Results

Three participants each completed all 15 Franka tasks; one user did Sukiyaki and the four Tiago tasks. Tasks took 1.83 attempts and 20.29 minutes on average, with about 80% of the time spent on human decision-making and decoding.
Task-time decoding accuracy: 81.2% for object selection, 42.2% for 4-way skill selection, 73.9% for 2-way parameter selection, and 100% for the jaw-clench confirmation. No task failed because of a decoding error.
Object-skill prediction reaches 0.94 accuracy on MakePasta (0.31 for a ResNet classifier) and cuts each object-skill selection from 45.7 to 18.1 s. Parameter prediction reduces mean cursor travel by 41% on SetTable.

Limitations

Decoding is slow, so tasks cannot involve time-sensitive or dynamic interaction, and interrupting the robot takes 500 ms. Failures came from users choosing a wrong skill or parameter and reaching an unrecoverable state. The skill library is limited, only three healthy subjects were tested, and the purely visual interface excludes people with severe visual impairments.

Videos

CleanBook: the Franka arm handles a book next to a wooden bookshelf.

CollectToy: the Franka arm picks toys off the table to put them in a basket.

Figures

From Figure 1: NOIR is a general-purpose brain-robot interface that allows humans to use their brain signals (1) to c... — From Figure 1: NOIR is a general-purpose brain-robot interface that allows humans to use their brain signals (1) to control robots to perform daily activities, such as making Sukiyaki (2), ironing clothes (7), playing Tic-Tac-Toe with friends (17), and petting a robot dog (21).

From Figure 2: NOIR has two components, a modular pipeline for decoding goals from human brain signals, and a robotic... — From Figure 2: NOIR has two components, a modular pipeline for decoding goals from human brain signals, and a robotic system with a library of primitive skills. The robots possess the ability to learn to predict human intended goals hence reducing the human effort required for decoding.

From Figure 3: A modular pipeline for decoding human intended goals from EEG signals: (a) What object to manipulate, ... — From Figure 3: A modular pipeline for decoding human intended goals from EEG signals: (a) What object to manipulate, decoded from SSVEP signals using CCA classifiers; (b) How to interact with the object, decoded from MI signals using CSP+QDA algorithms; (c) Where to interact, decoded from MI signals. A safety mechanism that captures muscle tension from jaw clench is used to confirm or reject decoding results.

From Figure 4: Retrieval-based few-shot object and skill selection model. The model learns a latent representation fo... — From Figure 4: Retrieval-based few-shot object and skill selection model. The model learns a latent representation for observations. Given a new observation, it finds the most relevant experience in the memory and selects the corresponding skill and object.

Abstract

We present Neural Signal Operated Intelligent Robots (NOIR), a general-purpose, intelligent brain-robot interface system that enables humans to command robots to perform everyday activities through brain signals. Through this interface, humans communicate their intended objects of interest and actions to the robots using electroencephalography (EEG). Our novel system demonstrates success in an expansive array of 20 challenging, everyday household activities, including cooking, cleaning, personal care, and entertainment. The effectiveness of the system is improved by its synergistic integration of robot learning algorithms, allowing for NOIR to adapt to individual users and predict their intentions. Our work enhances the way humans interact with robots, replacing traditional channels of interaction with direct, neural communication. Project website: https://noir-corl.github.io/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong