Differentiable MPC for End-to-end Planning and Control

Brandon Amos, Ivan Dario Jimenez Rodriguez, Jacob Sacks, Byron Boots, J. Zico Kolter

Seminal October 2018

TL;DR Make box-constrained MPC a differentiable policy layer by differentiating the convex approximation at the iLQR solver's fixed point, so the backward pass is a single extra LQR solve instead of an unrolled chain. Learning dynamics from expert actions alone matches system identification on pendulum and cartpole and beats it when the expert lies outside the model class.

What they do

The paper treats Model Predictive Control as a policy class whose cost and dynamics are learned end-to-end from a task loss. Earlier differentiable planners unrolled the optimizer; here the derivative is taken analytically at the solver’s fixed point.

How it works

Differentiable LQR. Implicitly differentiating LQR’s KKT conditions gives the gradients; the required linear solve is another LQR problem that reuses the forward-pass factorization.
Differentiable MPC. Nonlinear MPC with box-constrained controls is solved with box-DDP, which iterates convex approximations (quadratic cost, linear dynamics). At a fixed point, the final approximation is differentiated exactly with one LQR solve in which controls at their bounds are zeroed.
Training. Imitation learning on the loss between learner and expert action sequences, which needs only controls, not states.

Results

Against unrolling the iLQR solver (runtimes averaged over 10 trials), fixed-point differentiation is slightly faster in the forward pass and much faster in the backward pass, which costs one LQR solve instead of growing linearly with iLQR iterations.
LQR imitation (3-D state and control, horizon 5): of eight random initializations, half recover the true dynamics; the rest get stuck in local minima of the imitation loss.
Pendulum and cartpole: in nearly every case, learning cost, dynamics or both from actions significantly outperforms an LSTM policy trained on the same data, and learning dynamics from actions matches system identification, which uses states.
With an expert whose pendulum has extra damping and a “wind” force, system identification predicts next states slightly better but gives a worse model for imitation.

Limitations

The gradient assumes a fixed point; if the solver stops early or none exists, which often happens with neural-network dynamics, it is usually wrong and unrolling is needed. Learning cost weights and goal together was unstable. Experiments are simple simulated tasks with physics-parameterized dynamics.

Figures

From Figure 1: A learnable MPC module, with cost and dynamics submodules, inside a policy; gradients from the task lo... — From Figure 1: A learnable MPC module, with cost and dynamics submodules, inside a policy; gradients from the task loss on the actions flow back into the controller.

Abstract

We present foundations for using Model Predictive Control (MPC) as a differentiable policy class for reinforcement learning in continuous state and action spaces. This provides one way of leveraging and combining the advantages of model-free and model-based approaches. Specifically, we differentiate through MPC by using the KKT conditions of the convex approximation at a fixed point of the controller. Using this strategy, we are able to learn the cost and dynamics of a controller via end-to-end learning. Our experiments focus on imitation learning in the pendulum and cartpole domains, where we learn the cost and dynamics terms of an MPC policy class. We show that our MPC policies are significantly more data-efficient than a generic neural network and that our method is superior to traditional system identification in a setting where the expert is unrealizable.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong