Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images

Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, Martin Riedmiller

Seminal June 2015

TL;DR E2C trains a variational autoencoder whose latent dynamics are forced to be locally linear, so iLQR or AICO can plan directly on embeddings of raw images. It swings up a pendulum from 48x48 pixels in 90% of trials and balances a cart-pole and controls a three-link arm from images in 93% and 90%, where autoencoder baselines never succeed.

What they do

Locally optimal controllers like iLQG need a known model or a hand-designed low-dimensional state. Embed to Control (E2C) learns that state from images. Instead of fitting a latent space and then linearizing whatever dynamics result, it builds a locally linear transition into a VAE, so the latent space is shaped for linear-quadratic control. Training is unsupervised, on random (image, action, next image) tuples.

How it works

  • Encoder/decoder. A VAE maps each image to a diagonal Gaussian latent and back. Tasks where velocity matters stack two frames.
  • Locally linear transition. A network predicts A, B and an offset o from the latent, and the next latent is A z + B u + o. A is a rank-one perturbation of the identity.
  • Loss. The VAE bound on the current and next images, plus a KL term that pulls the predicted next latent toward the encoding of the real next image. This keeps multi-step predictions on valid encodings.
  • Control. The cost is quadratic distance to the encoded goal image. iLQR or AICO replans at every step with horizon T=10.

Results

  • Planar navigation (40x40 images, six obstacles): E2C reaches the goal from every start with a real trajectory cost of 25.1 (true model: 20.24). AE, VAE and VAE-with-slowness baselines succeed 0% of the time.
  • Pendulum swing-up (48x48 images): E2C succeeds 90% of the time with cost 15.4 (true model: 9.8). The non-linear-latent variant reaches 63.33%; Global E2C and E2C without the latent KL term reach 0%.
  • Cart-pole (80x80) and three-link arm (128x128), 8-D latent: 93% and 90% success, against 63% and 40% for non-linear E2C. Example trajectory costs are 11.13 and 85.12, versus 7.28 and 60.74 for AICO on the true dynamics.

Limitations

All four tasks are simulated, with rendered images. Control needs a goal image, and planar obstacles are encoded in advance with an added cost term. Render discretization makes small velocities hard to predict: uncontrolled cart-pole predictions drift right. Costs stay above planning on the true model.

Figures

From Figure 4: Left, a cart-pole trajectory in which only the first (green) frame is real and the rest are dreamed up...
From Figure 4: Left, a cart-pole trajectory in which only the first (green) frame is real and the rest are dreamed up by the model. Right, observed and predicted images along a robot-arm trajectory, with the goal marked in red.
From Figure 2: Latent spaces that an autoencoder, a VAE, a VAE with slowness and three E2C variants learn for the pla...
From Figure 2: Latent spaces that an autoencoder, a VAE, a VAE with slowness and three E2C variants learn for the planar system, made by embedding an image of every valid agent position.
From Figure 8: Latent spaces of all baseline models and E2C variants for the inverted pendulum.
From Figure 8: Latent spaces of all baseline models and E2C variants for the inverted pendulum.
Abstract

We introduce Embed to Control (E2C), a method for model learning and control of non-linear dynamical systems from raw pixel images. E2C consists of a deep generative model, belonging to the family of variational autoencoders, that learns to generate image trajectories from a latent space in which the dynamics is constrained to be locally linear. Our model is derived directly from an optimal control formulation in latent space, supports long-term prediction of image sequences and exhibits strong performance on a variety of complex control problems.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.