CLIPort: What and Where Pathways for Robotic Manipulation

Mohit Shridhar, Lucas Manuelli, Dieter Fox

Seminal September 2021

Franka Panda folding a cloth in half, then unfolding it, on command.

TL;DR CLIPort adds a frozen CLIP stream to the Transporter pick-and-place network, so one imitation-learned policy follows language instructions on tabletop tasks. In simulation it is the only model to average above 90% on seen tasks, and a single real-robot policy learns 9 tasks from 179 image-action pairs with 55-75% success.

What they do

Transporter places objects precisely but ignores language, so every new goal needs new demonstrations. CLIP grounds words in images but cannot say where to act. CLIPort combines them in a two-stream network that maps a top-down RGB-D image and an instruction to pick and place affordances, trained by imitation without object poses, segmentation or memory.

How it works

Action space. Each step is a pick pose and a place pose in SE(2). A pick network scores every pixel; a place network cross-correlates features of a crop around the pick with the full scene over 36 rotations.
Two streams. The spatial stream is Transporter’s ResNet on RGB-D, trained from scratch. The semantic stream decodes frozen CLIP ResNet50 features and multiplies them element-wise with the tiled CLIP sentence embedding at three decoder layers; lateral connections fuse in the spatial stream.
Training. Cross-entropy on one-hot demonstration actions with random SE(2) augmentation: 200K iterations for single-task models, 600K for multi-task.
Benchmark. 10 language-conditioned Ravens tasks (UR5e with suction), 8 with unseen-color, unseen-shape or unseen-object test splits.

Results

Average score on seen splits: Transporter-only saturates at 50% and CLIP-only at 76%. Only single-task CLIPort exceeds 90%, and it reaches 86% on most tasks with 100 demonstrations.
One multi-task model for all 10 tasks beats the single-task models in 41 of 72 evaluations. Training on other tasks’ unseen colors raises put-blocks-in-bowls-unseen-colors from 45.8 to 75.7 at 1000 demonstrations.
Unseen attributes are harder: stack-block-pyramid scores 98.8 on seen colors but 41.3 on unseen.
Real Franka Panda: one model for 9 tasks from 179 image-action pairs reaches 55-75% success, with 5-10 test runs per task.

Limitations

Each pick and place primitive runs open loop in SE(2), so tasks must be quasi-static, and execution depends on accurate camera-robot calibration. The model exploits dataset biases, cannot count, and struggles with spatial relations like “middle”. A human or oracle has to say when the task is done.

Videos

Packing named objects into a box, one instruction at a time.

Picking cherries one by one and putting them in a box.

Figures

From Figure 1: The 10 simulated Ravens tasks (a-j) and 5 of the 9 real-world Franka Panda tasks (k-o), each specified... — From Figure 1: The 10 simulated Ravens tasks (a-j) and 5 of the 9 real-world Franka Panda tasks (k-o), each specified by a language instruction.

From Figure 2: The two-stream architecture. The semantic stream encodes RGB with a frozen CLIP ResNet50 and condition... — From Figure 2: The two-stream architecture. The semantic stream encodes RGB with a frozen CLIP ResNet50 and conditions its decoder on tiled CLIP sentence features; the spatial stream encodes RGB-D and is fused into it laterally. The output is a dense pixelwise map for pick or place affordances.

From Figure 3: Average score over all tasks against number of training demonstrations, on seen (left) and unseen (rig... — From Figure 3: Average score over all tasks against number of training demonstrations, on seen (left) and unseen (right) splits.

Abstract

How can we imbue robots with the ability to manipulate objects precisely but also to reason about them in terms of abstract concepts? Recent works in manipulation have shown that end-to-end networks can learn dexterous skills that require precise spatial reasoning, but these methods often fail to generalize to new goals or quickly learn transferable concepts across tasks. In parallel, there has been great progress in learning generalizable semantic representations for vision and language by training on large-scale internet data, however these representations lack the spatial understanding necessary for fine-grained manipulation. To this end, we propose a framework that combines the best of both worlds: a two-stream architecture with semantic and spatial pathways for vision-based manipulation. Specifically, we present CLIPort, a language-conditioned imitation-learning agent that combines the broad semantic understanding (what) of CLIP [1] with the spatial precision (where) of Transporter [2]. Our end-to-end framework is capable of solving a variety of language-specified tabletop tasks from packing unseen objects to folding cloths, all without any explicit representations of object poses, instance segmentations, memory, symbolic states, or syntactic structures. Experiments in simulated and real-world settings show that our approach is data efficient in few-shot settings and generalizes effectively to seen and unseen semantic concepts. We even learn one multi-task policy for 10 simulated and 9 real-world tasks that is better or comparable to single-task policies.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong