Physical Interaction as Communication: Learning Robot Objectives Online from Human Corrections

Dylan P. Losey, Andrea Bajcsy, Marcia K. O'Malley, Anca D. Dragan

Seminal July 2021

TL;DR Treat a person's push on a robot arm as evidence about the objective the robot should optimize, and update that objective online so the robot changes the rest of its task after the person lets go. In a 10-person study on a 7-DoF arm, learning from corrections cut human effort, interaction time and task cost compared with impedance control (all p<0.0001).

What they do

Standard physical human-robot interaction (pHRI) controllers treat a push as a disturbance: the robot complies while touched, then returns to its original plan. This paper treats intentional pushes as corrections that reveal the objective the human wants. pHRI is formalized as a POMDP in which the weights of a linear feature reward are hidden and the human’s interaction torques are observations, and a real-time approximation learns those weights during the task and replans.

How it works

Approximations. Separate control from estimation (QMDP), keep only a MAP estimate of the weights, and plan at the trajectory level: replan an optimal trajectory for the current estimate and track it with impedance control.
Intended trajectory. Each push is propagated along the planned trajectory as a smooth deformation, giving the trajectory the human would prefer.
All-at-once update. Online gradient descent shifts the weights by the feature-count difference between the intended and planned trajectories.
One-at-a-time update. To limit unintended learning from noisy pushes, only the feature that changed most during the correction is updated.

Results

Simulation (2-DoF point robot): learning came close to a full QMDP solution (normalized regret 0.62 vs 0.55) and needed fewer corrections and less effort than only deforming the trajectory. A noisy simulated human corrected the All-at-Once robot during 5.24 timesteps on average versus 3.56 for One-at-a-Time.
User study 1 (10 people, 7-DoF JACO2, three cup-carrying tasks): Learning lowered interaction force, interaction time and task cost versus Impedance (all p<0.0001) and was rated more understanding, easier and more collaborative, but not more predictable.
User study 2 (12 people): One-at-a-Time made fewer weight updates in the wrong direction and learned a better final reward when all features needed correcting, but learned the table weight worse when only that one was wrong. Interaction time and force did not differ.

Limitations

Features are given by hand, and the studies used only two or three intuitive ones; larger or non-intuitive feature sets are untested. Preferences about timing would need a different way to infer the intended trajectory.

Videos

Figures

From Figure 2: One iteration of online learning. A point robot plans a straight line past two obstacles (black dotted... — From Figure 2: One iteration of online learning. A point robot plans a straight line past two obstacles (black dotted); the human pushes it, the push is deformed into a preferred trajectory (solid black), the objective is updated, and the replanned trajectory (orange dotted) keeps farther from both obstacles.

From Figure 17: The human means to teach the robot to move closer to the table, but the correction also tilts the cup... — From Figure 17: The human means to teach the robot to move closer to the table, but the correction also tilts the cup, so the robot must decide which parts of the correction to learn from and which to ignore.

From Figure 14: First user study. Learning from physical interaction decreased total participant effort compared with... — From Figure 14: First user study. Learning from physical interaction decreased total participant effort compared with Impedance on the cup, table and laptop tasks; an asterisk means p<.0001.

From Figure 20: Second user study. With One-at-a-Time, participants made fewer corrections that taught the robot the ... — From Figure 20: Second user study. With One-at-a-Time, participants made fewer corrections that taught the robot the opposite of what they intended on both tasks (left), and rated it less likely to learn the wrong thing and need undoing (right).

Abstract

When a robot performs a task next to a human, physical interaction is inevitable: the human might push, pull, twist, or guide the robot. The state-of-the-art treats these interactions as disturbances that the robot should reject or avoid. At best, these robots respond safely while the human interacts; but after the human lets go, these robots simply return to their original behavior. We recognize that physical human-robot interaction (pHRI) is often intentional -- the human intervenes on purpose because the robot is not doing the task correctly. In this paper, we argue that when pHRI is intentional it is also informative: the robot can leverage interactions to learn how it should complete the rest of its current task even after the person lets go. We formalize pHRI as a dynamical system, where the human has in mind an objective function they want the robot to optimize, but the robot does not get direct access to the parameters of this objective -- they are internal to the human. Within our proposed framework human interactions become observations about the true objective. We introduce approximations to learn from and respond to pHRI in real-time. We recognize that not all human corrections are perfect: often users interact with the robot noisily, and so we improve the efficiency of robot learning from pHRI by reducing unintended learning. Finally, we conduct simulations and user studies on a robotic manipulator to compare our proposed approach to the state-of-the-art. Our results indicate that learning from pHRI leads to better task performance and improved human satisfaction.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong