Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Socially Aware Motion Planning with Deep Reinforcement Learning

Yu Fan Chen, Michael Everett, Miao Liu, Jonathan P. How

Seminal March 2017

TL;DR SA-CADRL trains a deep-RL collision-avoidance value network with a small penalty for breaking a social norm, such as passing on the wrong side, and extends it to four agents. The right-handed policy picks the norm-respecting side in 83-98% of two-agent test cases, at about 20% more extra time to goal. It drove a robot fully autonomously among pedestrians at 1.2 m/s.

What they do

Socially compliant navigation is hard to specify directly, because people’s paths vary from person to person and from run to run. The paper argues that it is easier to specify what not to do. It builds on CADRL, a deep-RL method for cooperative collision avoidance. SA-CADRL adds a penalty for violating a chosen norm (e.g. pass on the right, overtake on the left) and trains on multiagent scenarios directly.

How it works

  • Value network. An agent’s state and the observable state of its neighbors (position, velocity, radius) are expressed in a goal-aligned local frame. The network maps them to the expected time to goal, and the policy picks the action with the best value.
  • Norm-inducing reward. A scalar penalty applies when another agent is in a region that means the wrong side for passing, crossing or overtaking. The region need not be precise: the penalty only breaks the tie between equally fast left- and right-handed solutions.
  • Multiagent network. Weight sharing over neighbor inputs plus max-pooling makes the value independent of neighbor order. A four-agent network handles fewer agents by filling the empty slots with flagged virtual agents.
  • Training. A separate set of bad (collision) experiences focuses learning, because at least 90% of random cases were nearly straight-line. Trajectories are also mirrored in the x-axis, so the policy explores the other passing topology.

Results

  • Two agents, 200 random cases: right-handed SA-CADRL takes the right-handed side in 98% of passing, 85% of crossing and 83% of overtaking cases. ORCA is about 50-50. Its average extra time to goal is 0.31 s, against 0.46 s for ORCA and 0.25 s for CADRL.
  • Four agents: the RL methods need 0.41-0.49 s of extra time, against 0.86 s for ORCA. The 10th-percentile separation is 0.155 m for SA-CADRL and 0.096 m for CADRL (minimax).
  • Each value-network query takes 8.7 ms on an i7 CPU. Training took about 9 h for 3,000 episodes.
  • Hardware: ten runs to random goals more than 50 m apart, at 1.2 m/s, with an average of 10.2 people per minute passing within 2 m. The safety driver never intervened.

Limitations

Norm compliance trades off against time to goal, and the paper sets this tradeoff with the penalty weight. Larger penalty regions could make training unstable or diverge. The hardware results are reported qualitatively. Groups of pedestrians walking together are not modeled and are left to future work.

Videos

Full video (YouTube)

Figures

From Figure 1: A robotic vehicle navigating autonomously in a pedestrian-rich environment.
From Figure 1: A robotic vehicle navigating autonomously in a pedestrian-rich environment.
From Figure 6: Two agents crossing under the SA-CADRL policy trained for right-handed rules. Circles show each agent'...
From Figure 6: Two agents crossing under the SA-CADRL policy trained for right-handed rules. Circles show each agent's position at the labeled time, and stars mark the goals.
From Figure 7: Four agents moving to the opposite side under the four-agent SA-CADRL policy with right-handed rules. ...
From Figure 7: Four agents moving to the opposite side under the four-agent SA-CADRL policy with right-handed rules. Circles show each agent's position at the labeled time, and stars mark the goals.
From Figure 8: The robot's sensor data as the vehicle (yellow box) passes a pedestrian on the right using the right-h...
From Figure 8: The robot's sensor data as the vehicle (yellow box) passes a pedestrian on the right using the right-handed SA-CADRL policy at a nominal 1.2 m/s.
Abstract

For robotic vehicles to navigate safely and efficiently in pedestrian-rich environments, it is important to model subtle human behaviors and navigation rules (e.g., passing on the right). However, while instinctive to humans, socially compliant navigation is still difficult to quantify due to the stochasticity in people's behaviors. Existing works are mostly focused on using feature-matching techniques to describe and imitate human paths, but often do not generalize well since the feature values can vary from person to person, and even run to run. This work notes that while it is challenging to directly specify the details of what to do (precise mechanisms of human navigation), it is straightforward to specify what not to do (violations of social norms). Specifically, using deep reinforcement learning, this work develops a time-efficient navigation policy that respects common social norms. The proposed method is shown to enable fully autonomous navigation of a robotic vehicle moving at human walking speed in an environment with many pedestrians.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.