Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Autonomy Infused Teleoperation with Application to BCI Manipulation

Katharina Muelling, Arun Venkatraman, Jean-Sebastien Valois, John Downey, Jeffrey Weiss, Shervin Javdani, Martial Hebert, Andrew B. Schwartz, Jennifer L. Collinger, J. Andrew Bagnell

Seminal March 2015

TL;DR A shared-control system recognizes objects, infers which grasp the user is reaching for and blends the user's command with an autonomous one, so two people with intracortical brain-computer interfaces can control a 7-DoF arm. On adapted Action Research Arm Test tasks, the subjects completed none of their 15 trials each under direct control, versus 13/15 and 8/15 with assistance.

What they do

Commands decoded from an intracortical brain-computer interface are noisy and low-dimensional: here 3D hand translation plus a grasp signal. The system recognizes objects, infers which grasp the user is reaching for and blends its own motion with the user’s. Two tetraplegic subjects used it to control a 7-DoF Barrett WAM with a BarrettHand.

How it works

  • Perception. Objects are found by matching RGB-D depth images to templates rendered from laser-scanned models, and ICP refines each pose.
  • Capture envelopes. Each stored grasp has a pose and an approach direction. A cone along that direction measures progress, so the hand is never pulled backward into the object. Rotation difference and kinematic feasibility add to the cost, and the cheapest grasp becomes the robot’s goal.
  • Intent and arbitration. With several objects, maximum-entropy inference scores each goal from the trajectory so far. The robot and user commands are blended linearly. The user’s share is a sigmoid of confidence, bounded below by a per-user minimum, and a large disagreement lets the user break away.
  • Compliance. A wrist force-torque sensor drives Jacobian-transpose joint offsets, and stall detection relaxes the joints.

Results

  • On adapted Action Research Arm Test tasks, neither subject completed a trial under direct control (0/15 each). With assistance, Subject 1 completed 13/15 and Subject 2 completed 8/15 (Subject 2 grasped in 13).
  • Perceived difficulty (1-10) fell from 8.8 to 2.4 for Subject 1 and from 7.8 to 3.2 for Subject 2.
  • Box and Blocks: on average 5.2 transfers in two minutes with assistance versus 2.6 under direct control.
  • Choosing between two blocks, Subject 2 grasped the indicated one 32/36 times; intent inference was wrong once.

Limitations

Grasping relies on known object models with predefined grasps; novel objects in clutter were only a proof of concept, and pouring and that demo used a game controller rather than the brain-computer interface. Only two subjects were tested, with three trials per object and mode. In one session the arm blocked the camera, which left the Box and Blocks results inconclusive.

Figures

From Figure 1: A subject with a brain-computer interface sits beside the table and teleoperates the robot arm and hand.
From Figure 1: A subject with a brain-computer interface sits beside the table and teleoperates the robot arm and hand.
From Figure 2: The assistance module (green) infers the user's intent from their end-effector and grasp velocities an...
From Figure 2: The assistance module (green) infers the user's intent from their end-effector and grasp velocities and a model of the scene, plans an autonomous motion, and blends it with the user's input into arm and finger joint velocities for a compliant controller (red).
From Figure 5: End-effector path while moving the 5 cm block to the release target with Autonomy Infused Teleoperatio...
From Figure 5: End-effector path while moving the 5 cm block to the release target with Autonomy Infused Teleoperation, from start (green) to end (red); the brown dot marks the object.
From Figure 5: The same 5 cm block task under direct control, where the path wanders and the object (brown dots) gets...
From Figure 5: The same 5 cm block task under direct control, where the path wanders and the object (brown dots) gets pushed around and never reaches the target.
Abstract

Robot teleoperation systems face a common set of challenges including latency, low-dimensional user commands, and asymmetric control inputs. User control with Brain-Computer Interfaces (BCIs) exacerbates these problems through especially noisy and erratic low-dimensional motion commands due to the difficulty in decoding neural activity. We introduce a general framework to address these challenges through a combination of computer vision, user intent inference, and arbitration between the human input and autonomous control schemes. Adjustable levels of assistance allow the system to balance the operator's capabilities and feelings of comfort and control while compensating for a task's difficulty. We present experimental results demonstrating significant performance improvement using the shared-control assistance framework on adapted rehabilitation benchmarks with two subjects implanted with intracortical brain-computer interfaces controlling a seven degree-of-freedom robotic manipulator as a prosthetic. Our results further indicate that shared assistance mitigates perceived user difficulty and even enables successful performance on previously infeasible tasks. We showcase the extensibility of our architecture with applications to quality-of-life tasks such as opening a door, pouring liquids from containers, and manipulation with novel objects in densely cluttered environments.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.