Shared Autonomy via Hindsight Optimization

Shervin Javdani, Siddhartha S. Srinivasa, J. Andrew Bagnell

★ Must read March 2015

TL;DR Shared autonomy as a POMDP over the user's unknown goal, solved approximately with hindsight optimization so the robot assists toward all likely goals at once. In a 10-user grasping study, users were significantly faster (p=0.006) and used less input than with predict-then-blend, yet trended toward preferring the blend baseline.

What they do

In shared autonomy the robot often does not know which object the user wants. Instead of predicting the most likely goal and assisting toward it (predict-then-blend), this paper treats assistance as a POMDP over the user’s goal and minimizes expected cost-to-go across the whole goal distribution, so the robot can help before any single goal is confidently predicted.

How it works

Goal inference. The user is modeled with maximum entropy inverse optimal control as approximately optimizing a cost for their goal; Bayes’ rule over the input history gives a distribution over goals. Probabilities come from user inputs only, not robot actions, so the robot cannot reinforce its own beliefs.
Hindsight optimization (QMDP). Cost-to-go is approximated by assuming the goal becomes known at the next step, so the robot never acts just to gather information. Assuming the robot then finishes alone makes per-goal values analytic and needs no user-policy model.
Multiple targets per goal. Each object has several acceptable grasp poses; goal value functions decompose over targets.
Costs. The robot’s cost is a distance-based user cost plus a squared penalty for deviating from the direct teleoperation command.

Results

Ten users teleoperated a robot arm with two Razer Hydra joysticks to grasp a canteen, block or cup, using the method (policy) and a predict-then-blend baseline (blend).

Policy made users faster (F(1,9)=12.98, p=0.006) and reduced total input (F(1,9)=7.76, p=0.021).
Preference did not favor policy: “would like to use” averaged 4.90 for blend vs 4.10 for policy, and the preference question averaged 2.90 (1 = blend, 7 = policy); neither was significant.
Users felt more in control with blend, and some who preferred blend were seen opposing the robot’s actions.

Limitations

The user cost is a simple distance function without collision avoidance, and the model assumes users do not adapt to assistance. The study had 10 participants, all with robotics experience, who cited a trade-off between control authority and speed.

Figures

From Figure 1: The shared autonomy framework. The user follows a policy for their own goal (here the canteen, shown a... — From Figure 1: The shared autonomy framework. The user follows a policy for their own goal (here the canteen, shown as a value-function heatmap) without knowledge of assistance; the system infers a distribution over all goals from user inputs and uses each goal's value function to choose the robot action.

From Figure 3: Goal probabilities and a 2D slice of the estimated value function during a grasping trial, after the u... — From Figure 3: Goal probabilities and a 2D slice of the estimated value function during a grasping trial, after the user inputs "left". As one object's probability dominates, the system automatically rotates the end-effector to grasp it.

From Figure 6: Mean task completion time with standard errors for the predict-then-blend baseline (blend) and the pro... — From Figure 6: Mean task completion time with standard errors for the predict-then-blend baseline (blend) and the proposed method (policy).

From Figure 7: Mean survey ratings with standard errors on a 1-7 scale: felt in control, robot did what I wanted, acc... — From Figure 7: Mean survey ratings with standard errors on a 1-7 scale: felt in control, robot did what I wanted, accomplished tasks quickly, would like to use the system, and which system they prefer (1 = blend, 7 = policy).

Abstract

In shared autonomy, user input and robot autonomy are combined to control a robot to achieve a goal. Often, the robot does not know a priori which goal the user wants to achieve, and must both predict the user's intended goal, and assist in achieving that goal. We formulate the problem of shared autonomy as a Partially Observable Markov Decision Process with uncertainty over the user's goal. We utilize maximum entropy inverse optimal control to estimate a distribution over the user's goal based on the history of inputs. Ideally, the robot assists the user by solving for an action which minimizes the expected cost-to-go for the (unknown) goal. As solving the POMDP to select the optimal action is intractable, we use hindsight optimization to approximate the solution. In a user study, we compare our method to a standard predict-then-blend approach. We find that our method enables users to accomplish tasks more quickly while utilizing less input. However, when asked to rate each system, users were mixed in their assessment, citing a tradeoff between maintaining control authority and accomplishing tasks quickly.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong