Time-Contrastive Networks: Self-Supervised Learning from Video

Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine

Seminal April 2017

A KUKA arm pours beads after learning from one video of a human pouring a drink.

TL;DR Learn an image embedding from unlabeled multi-view video by pulling together simultaneous frames from different cameras and pushing apart nearby frames in time. Used as a reward, it lets a real KUKA arm learn to pour beads from a single video of a human pouring in 10 RL iterations.

What they do

Imitating a human from video needs features that ignore viewpoint and embodiment but track what changes during a task, such as hand contact, container tilt, liquid level or body pose. Time-Contrastive Networks (TCN) learn such an embedding without labels, from videos filmed simultaneously by several cameras (e.g. two smartphones). The embedding then serves as an RL reward for imitating a single 3rd-person human video, and for real-time pose imitation.

How it works

Loss. A triplet loss pulls together the same moment seen from two viewpoints and pushes away a temporal neighbor from the same sequence, which looks similar but is functionally different.
Network. ImageNet-pretrained Inception layers, two conv layers and a spatial softmax, ending in a 32-dimensional embedding.
Reward. Squared Euclidean plus Huber-style distance between the TCN embeddings of the human video and the robot’s camera frames at each time step, optimized with PILQR. The TCN features join joint angles and velocities in the state.
Pose imitation. A TCN trained on humans and a robot moving randomly gets a decoder that regresses the robot’s 8 joints from images of the robot itself; it is then applied to human images.

Results

Pouring test videos: multi-view TCN reaches 18.0-18.8% alignment error and 19.6-22.2% attribute classification error, against 22.8% / 27.0% for shuffle & learn and 29.8% / 51.9% for ImageNet Inception features.
Real robot: with a TCN trained on about 1 hour of pouring-related video, a KUKA arm learns to pour beads from one human video in 10 iterations of 10 rollouts. Rewards from single-view TCN, shuffle & learn and Inception fail. A simulated dish-rack plate transfer is also learned in 10 iterations.
Pose imitation joint error: 32.1% with no human labels (TC+Self), against 33.4% for human-supervised regression and 29.5% with all signals combined.

Limitations

Training needs multi-viewpoint video; the single-view variant does substantially worse. Each task uses its own TCN. Relating human to robot pouring required about 20 minutes of robot-arm footage. The label-free pose model gets the shoulder pan joint wrong, doing worse than random on it.

Videos

The pouring policy improving over RL iterations, rewarded by TCN distance to the human video.

A Fetch robot imitates a human's pose from video, trained without pose labels.

Figures

From Figure 1: Anchor and positive frames taken at the same time from different viewpoints are pulled together in the... — From Figure 1: Anchor and positive frames taken at the same time from different viewpoints are pulled together in the embedding, while negative frames from a different time in the same sequence are pushed away. The model learns what is common between the different-looking blue frames and what differs between the similar-looking red and blue frames.

From Figure 7: Weight of beads poured into the target cup (maximum 189 g) per PILQR iteration, learned from a single ... — From Figure 7: Weight of beads poured into the target cup (maximum 189 g) per PILQR iteration, learned from a single 3rd-person human demonstration, with the standard deviation over 10 rollouts per iteration. The robot learns the task with the multi-view TCN reward after 10 iterations.

From Figure 10: Nearest neighbors (right) of each reference frame (left) under a multi-view TCN trained without human... — From Figure 10: Nearest neighbors (right) of each reference frame (left) under a multi-view TCN trained without human labels. It matches poses such as crouching and reaching up across humans and robots, viewpoints, backgrounds and scale.

Abstract

We propose a self-supervised approach for learning representations and robotic behaviors entirely from unlabeled videos recorded from multiple viewpoints, and study how this representation can be used in two robotic imitation settings: imitating object interactions from videos of humans, and imitating human poses. Imitation of human behavior requires a viewpoint-invariant representation that captures the relationships between end-effectors (hands or robot grippers) and the environment, object attributes, and body pose. We train our representations using a metric learning loss, where multiple simultaneous viewpoints of the same observation are attracted in the embedding space, while being repelled from temporal neighbors which are often visually similar but functionally different. In other words, the model simultaneously learns to recognize what is common between different-looking images, and what is different between similar-looking images. This signal causes our model to discover attributes that do not change across viewpoint, but do change across time, while ignoring nuisance variables such as occlusions, motion blur, lighting and background. We demonstrate that this representation can be used by a robot to directly mimic human poses without an explicit correspondence, and that it can be used as a reward function within a reinforcement learning algorithm. While representations are learned from an unlabeled collection of task-related videos, robot behaviors such as pouring are learned by watching a single 3rd-person demonstration by a human. Reward functions obtained by following the human demonstrations under the learned representation enable efficient reinforcement learning that is practical for real-world robotic systems. Video results, open-source code and dataset are available at https://sermanet.github.io/imitate

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong