Efficient Model Learning for Human-Robot Collaborative Tasks

Stefanos Nikolaidis, Keren Gu, Ramya Ramakrishnan, Julie Shah

Seminal May 2014

TL;DR Cluster human-team demonstrations into a few "human types", learn a reward per type with inverse RL, and plan with a MOMDP that treats the type of a new partner as hidden. Clustering matches an expert's labels 96.5% of the time on a place-and-drill task, and the policy holds its reward as human actions deviate from the demonstrations.

What they do

A robot working next to a person should adapt to their style, but learning a new model for every user is slow. Observing that human teams follow only a few high-level strategies, the authors learn a small set of “dominant” human types from unlabeled team demonstrations. A new user’s type is a hidden variable the robot infers, offline from a few demonstrations or online during the task.

How it works

Clustering. Each demonstration is a sequence of alternating human and robot actions. A hard-EM algorithm represents each cluster by an action-transition matrix; BIC picks the number of clusters k from 2 to 10, with 20 EM restarts per k.
Reward learning. For each cluster, inverse RL (Abbeel and Ng’s feature-matching method) learns a reward over task steps.
Planning. A Mixed Observability MDP has the task step as the observable state and the human type as the hidden state. SARSOP computes a policy that maximizes expected reward under the belief over types.

Results

Place-and-drill task, 18 subjects, 54 sequences, leave-one-out: clustering finds two types, “safe” (place all screws before drilling) and “efficient” (drill each screw right after placing it), and agrees with an expert’s hand labels 96.5% of the time.
As the simulated human takes random actions more often, the cross-training baseline’s reward drops while the MOMDP policy’s stays roughly flat; it performs comparably to a MOMDP hand-coded by a domain expert.
Hand-finishing task (6 subjects, 2000 states): the robot estimates the worker’s preferred side online from motion-capture hand position, waits while unsure, and then presents that side of the box.

Limitations

Test subjects mostly repeated their demonstrated actions, so robustness to deviations was only tested in simulation. A person’s type is assumed constant within a demonstration, and clustering worked well partly because the two types were clearly separated. In the hand-finishing task the observation model was set by hand, not learned.

Figures

From Figure 1: Framework flowchart. Offline, demonstrated sequences are clustered, a reward function is learned per c... — From Figure 1: Framework flowchart. Offline, demonstrated sequences are clustered, a reward function is learned per cluster and a MOMDP policy is computed; online, the new user's type is inferred while human and robot do the task.

From Figure 4: Task execution by a human-robot team on the place-and-drill task.

From Figure 5: Accumulated reward, averaged over 18 cross-validation folds, for the learned MOMDP policy, a MOMDP han... — From Figure 5: Accumulated reward, averaged over 18 cross-validation folds, for the learned MOMDP policy, a MOMDP hand-coded by a domain expert and human-robot cross-training. The x-axis is the probability that the simulated human takes a random action instead of the one they actually took.

From Figure 6: Hand-finishing task with an industrial robot. The worker first moves right and the robot stays still; ... — From Figure 6: Hand-finishing task with an industrial robot. The worker first moves right and the robot stays still; when the worker moves left, the robot moves the box to the left and then rotates it so the worker can refinish the surface.

Abstract

We present a framework for learning human user models from joint-action demonstrations that enables the robot to compute a robust policy for a collaborative task with a human. The learning takes place completely automatically, without any human intervention. First, we describe the clustering of demonstrated action sequences into different human types using an unsupervised learning algorithm. These demonstrated sequences are also used by the robot to learn a reward function that is representative for each type, through the employment of an inverse reinforcement learning algorithm. The learned model is then used as part of a Mixed Observability Markov Decision Process formulation, wherein the human type is a partially observable variable. With this framework, we can infer, either offline or online, the human type of a new user that was not included in the training set, and can compute a policy for the robot that will be aligned to the preference of this new user and will be robust to deviations of the human actions from prior demonstrations. Finally we validate the approach using data collected in human subject experiments, and conduct proof-of-concept demonstrations in which a person performs a collaborative task with a small industrial robot.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong