Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning

Changan Chen, Yuejiang Liu, Sven Kreiss, Alexandre Alahi

Seminal September 2018

TL;DR A deep RL value network for crowd navigation that encodes every robot-human pair, adds a local map of each human's neighbors, and pools the pairs with self-attention. In simulated crowds that ignore the robot it reaches the goal in all 500 test cases, versus 0.95 for an LSTM baseline and 0.78 for CADRL.

What they do

Earlier RL crowd-navigation policies aggregate the crowd crudely (CADRL keeps one worst-case human via maximin; LSTM-RL feeds humans in order of distance) and model only how humans affect the robot. This paper learns a value network that encodes every robot-human pair, includes a coarse model of human-human interaction, and weights each human with a learned self-attention score.

How it works

  • Interaction module. In a robot-centric frame, an MLP embeds each human’s state, the robot’s state and a 4x4 local map (1 m cells holding the presence and velocity of that human’s neighbors), then a second MLP gives a pairwise interaction feature. The map avoids explicit O(N^2) human-human modeling.
  • Attentive pooling. An MLP scores each human from its embedding plus the mean embedding of all humans. The weighted sum of interaction features gives a fixed-length crowd vector for any number of humans.
  • Planning. An MLP maps the crowd vector to a state value; actions come from 80 discrete holonomic velocities (5 speeds x 16 headings).
  • Training. Imitation warm start on 3k ORCA episodes, then 10k episodes of temporal-difference learning (about 10 h on a CPU) against ORCA-driven simulated humans.

Results

  • Robot invisible to humans (500 test cases): SARL and LM-SARL succeed 1.00, versus 0.95 for LSTM-RL, 0.78 for CADRL and 0.43 for ORCA. LM-SARL navigates in 10.46 s versus 11.82 s for LSTM-RL.
  • Robot visible: LM-SARL succeeds 1.00 in 10.59 s (reward 0.334). ORCA succeeds 0.99 but takes 12.29 s (reward 0.284); CADRL collides in 0.03 of cases.
  • The local map changes reward by under 0.01, since human-human interactions matter only some of the time; qualitatively, only LM-SARL’s value map favors cutting behind humans 1 and 3 when human 3 might turn toward the robot.

Limitations

Assumes a holonomic robot that reaches commanded velocities instantly, and fully observed human positions, velocities and radii. Training queries the simulator for the true next state; deployment would need a trajectory predictor. All numbers come from one simulated circle-crossing scenario with ORCA humans; the Segway robot test is shown only as a video.

Videos

Full video (YouTube)

Figures

From Figure 1: The robot models both Human-Robot and Human-Human interactions to navigate through a crowd.
From Figure 1: The robot models both Human-Robot and Human-Human interactions to navigate through a crowd.
From Figure 2: Three modules. The interaction module extracts robot-human interactions, the pooling module aggregates...
From Figure 2: Three modules. The interaction module extracts robot-human interactions, the pooling module aggregates them, and the planning module estimates the value of the joint robot-crowd state.
From Figure 4: The pooling module. An MLP scores each person from their embedding and the mean embedding; the crowd r...
From Figure 4: The pooling module. An MLP scores each person from their embedding and the mean embedding; the crowd representation is the weighted sum of the pairwise interaction features.
From Figure 6: Attention scores in a dense scene. LM-SARL gives low scores to humans 4 and 5, who walk away, and the ...
From Figure 6: Attention scores in a dense scene. LM-SARL gives low scores to humans 4 and 5, who walk away, and the highest to human 3, who is most likely to get close soon.
Abstract

Mobility in an effective and socially-compliant manner is an essential yet challenging task for robots operating in crowded spaces. Recent works have shown the power of deep reinforcement learning techniques to learn socially cooperative policies. However, their cooperation ability deteriorates as the crowd grows since they typically relax the problem as a one-way Human-Robot interaction problem. In this work, we want to go beyond first-order Human-Robot interaction and more explicitly model Crowd-Robot Interaction (CRI). We propose to (i) rethink pairwise interactions with a self-attention mechanism, and (ii) jointly model Human-Robot as well as Human-Human interactions in the deep reinforcement learning framework. Our model captures the Human-Human interactions occurring in dense crowds that indirectly affects the robot's anticipation capability. Our proposed attentive pooling mechanism learns the collective importance of neighboring humans with respect to their future states. Various experiments demonstrate that our model can anticipate human dynamics and navigate in crowds with time efficiency, outperforming state-of-the-art methods.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.