Shared Autonomy via Deep Reinforcement Learning

Siddharth Reddy, Anca D. Dragan, Sergey Levine

Seminal February 2018

TL;DR A deep Q-learning copilot reads the user's raw control input as an observation and executes the near-optimal action closest to it, so it can assist without knowing the dynamics, the user's policy or the goal. In a 12-person Lunar Lander study, every participant succeeded more often and crashed less with the copilot (p < 0.0001).

What they do

Earlier shared-autonomy methods infer the user’s goal, but they assume known dynamics, a known set of goals and a known model of how the user acts. This paper drops all three. A model-free deep Q-learning “copilot” treats the user’s control input as one more observation and learns from task reward alone to help with a goal it never observes.

How it works

Input. The environment state is concatenated with the user’s raw action. If the goal space is known, an LSTM goal predictor or Bayesian goal inference can supply the input instead.
Control sharing. The copilot discards actions whose value falls too far below the best one, then executes the remaining action closest to the user’s command. The tolerance α ∈ [0,1] sets how much assistance it gives.
Training. Neural fitted Q-iteration, which updates only at episode end so the interface does not lag. Reward combines hand-coded terms (penalties for speed and tilt, or for distance to the pad) with a terminal success/crash signal. The copilot is pretrained without a pilot, then fine-tuned with the human.

Results

Simulated Lunar Lander pilots: the copilot raises a laggy pilot’s success rate from 0.15 to 0.287 and cuts its crash rate from 0.75 to 0.186. For a noisy pilot the same rates go from 0.15/0.70 to 0.24/0.16.
When a user model is available, Bayesian goal inference beats both supervised goal prediction and raw action input.
Lunar Lander study (n = 12): every participant had more successes and fewer crashes with the copilot (ANOVA F(1,11) = 165.0 for success and 260.0 for crashes, p < 0.0001).
Quadrotor perching (n = 4, AR-Drone 2): the user aims the camera at an object the agent cannot see. Assisted pilots reliably reached the pad (F(1,3) = 44.1 and 62.3, p < 0.01).

Limitations

The copilot has no memory of past inputs. The drone study is small, and the authors call its evidence mainly anecdotal. Lunar Lander had to be made easier for humans, and α was set by hand. Assisted teams crashed more often than the solo copilot. Users found the copilot somewhat inconsistent and too aggressive in stabilizing flight.

Figures

From Figure 1: Overview. The state and the user's control input go into a Q-network that picks the executed action, a... — From Figure 1: Overview. The state and the user's control input go into a Q-network that picks the executed action, and the user gives success or failure feedback. Test beds: Lunar Lander (a) and a real quadrotor flown through its onboard camera (b, c).

From Figure 3: Lunar Lander user study (n = 12). Success and crash rates over 30 episodes for each human pilot alone ... — From Figure 3: Lunar Lander user study (n = 12). Success and crash rates over 30 episodes for each human pilot alone and with the copilot, and for the copilot alone.

From Figure 3: Quadrotor perching user study (n = 4). Success and crash rates over 20 episodes for each human pilot a... — From Figure 3: Quadrotor perching user study (n = 4). Success and crash rates over 20 episodes for each human pilot alone and with the copilot, and for the copilot alone.

Abstract

In shared autonomy, user input is combined with semi-autonomous control to achieve a common goal. The goal is often unknown ex-ante, so prior work enables agents to infer the goal from user input and assist with the task. Such methods tend to assume some combination of knowledge of the dynamics of the environment, the user's policy given their goal, and the set of possible goals the user might target, which limits their application to real-world scenarios. We propose a deep reinforcement learning framework for model-free shared autonomy that lifts these assumptions. We use human-in-the-loop reinforcement learning with neural network function approximation to learn an end-to-end mapping from environmental observation and user input to agent action values, with task reward as the only form of supervision. This approach poses the challenge of following user commands closely enough to provide the user with real-time action feedback and thereby ensure high-quality user input, but also deviating from the user's actions when they are suboptimal. We balance these two needs by discarding actions whose values fall below some threshold, then selecting the remaining action closest to the user's input. Controlled studies with users (n = 12) and synthetic pilots playing a video game, and a pilot study with users (n = 4) flying a real quadrotor, demonstrate the ability of our algorithm to assist users with real-time control tasks in which the agent cannot directly access the user's private information through observations, but receives a reward signal and user input that both depend on the user's intent. The agent learns to assist the user without access to this private information, implicitly inferring it from the user's input. This paper is a proof of concept that illustrates the potential for deep reinforcement learning to enable flexible and practical assistive systems.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong