Affordances from Human Videos as a Versatile Representation for Robotics

Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, Deepak Pathak

Seminal April 2023

Human videos, the learned affordance predictions and robot execution, side by side.

TL;DR VRB learns from EPIC-Kitchens human videos where a person would touch a scene and how their hand moves afterwards, then uses these predictions to drive four kinds of robot learning. With imitation from data collected this way, real robots average 57% success across 8 tasks, against 25% for the next-best affordance model.

What they do

Affordance models trained on human video had been tested on video datasets but not on robots. VRB (Vision-Robotics Bridge) represents an affordance in a form a robot can execute: a contact point and a post-contact trajectory, both in pixel space. A model trained on EPIC-Kitchens predicts them from one image of a scene with no human in it. Its outputs plug into four robot learning setups: data collection for imitation, reward-free exploration, goal-conditioned learning and a discrete action space for RL. Experiments use a Franka Panda in two play kitchens and a Hello Stretch on in-the-wild tasks (door, drawer, lid, garbage can, dishwasher).

How it works

Labels from video. A hand-object detector finds the first contact frame. Skin segmentation gives contact points around the hand, which are fitted with a 5-component GMM, and the hand box track after contact gives the trajectory. Homographies remove camera motion, and labels are mapped back to the frame before the hand entered, so the model never sees humans. About 54K training tuples.
Model. A ResNet18 encoder feeds deconvolution layers that output 5 contact heatmaps and a transformer that predicts 5 relative waypoints, from 150x150 crops.
On the robot. Predictions are lifted to 3D with a calibrated RealSense. The robot picks a gripper rotation, moves to the contact point, grasps, then follows the waypoints.
Learning on top. Exploration and goal reaching refit a distribution to the best trajectories (most scene change, or closest to a goal image) and resample. For RL, GMMs over many predictions give a 4x4 discrete action space for a DQN.

Results

Imitation (k-NN and behavior cloning on 30-150 collected episodes per task): VRB wins on 7 of 8 tasks, averaging 57% success against 25% for Hotspots.
Exploration: 3x to 10x more coincidental successes than HAP and random sampling on every task. On cabinet opening, VRB has about 2x the successes and over 6x the partial successes of the baselines.
Simulated Franka-Kitchen data collection: 0.75 / 0.60 / 0.55 success on light, microwave and kettle, against at best 0.60 / 0.45 / 0.45 for baselines.
As a frozen encoder for behavior cloning, VRB beats R3M on all three simulated tasks (slide-door 0.84 vs 0.70).

Limitations

Tasks are single-stage grasp-then-move primitives with a few discrete gripper rotations; the authors leave multi-stage tasks and force or tactile information to future work. Success is judged by manual inspection. Some tasks stay low (knife 0.3, lid 0.4 with k-NN). The RL action space is defined for one initial image, so the scene must be reset close to it.

Videos

The model predicts contact-point heatmaps and an interaction trajectory on a kitchen cabinet.

A Hello Stretch robot reaches for and grasps the lid handle of a pot.

Figures

From Figure 1: Visual affordances learned from human videos, deployed on real robots in the wild across several tasks... — From Figure 1: Visual affordances learned from human videos, deployed on real robots in the wild across several tasks and learning paradigms.

From Figure 2: VRB overview. The model learns from human videos to predict contact points and trajectory waypoints, s... — From Figure 2: VRB overview. The model learns from human videos to predict contact points and trajectory waypoints, supervised by future frames; on the robot, its outputs are converted to 3D actions.

From Figure 3: The four robot learning paradigms: (a) offline data collection for imitation, (b) exploration with int... — From Figure 3: The four robot learning paradigms: (a) offline data collection for imitation, (b) exploration with intrinsic rewards, (c) goal-conditioned learning with a goal image as reward, (d) affordances as a reduced, discrete action space.

From Figure 4: Predicted contact regions and post-grasp trajectories on the same scenes for VRB (green arrows), HOI (... — From Figure 4: Predicted contact regions and post-grasp trajectories on the same scenes for VRB (green arrows), HOI (red), Hotspots and HAP.

Abstract

Building a robot that can understand and learn to interact by watching humans has inspired several vision problems. However, despite some successful results on static datasets, it remains unclear how current models can be used on a robot directly. In this paper, we aim to bridge this gap by leveraging videos of human interactions in an environment centric manner. Utilizing internet videos of human behavior, we train a visual affordance model that estimates where and how in the scene a human is likely to interact. The structure of these behavioral affordances directly enables the robot to perform many complex tasks. We show how to seamlessly integrate our affordance model with four robot learning paradigms including offline imitation learning, exploration, goal-conditioned learning, and action parameterization for reinforcement learning. We show the efficacy of our approach, which we call VRB, across 4 real world environments, over 10 different tasks, and 2 robotic platforms operating in the wild. Results, visualizations and videos at https://robo-affordances.github.io/

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong