PaLM-E: An Embodied Multimodal Language Model

Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, Pete Florence

Seminal March 2023

Mobile robot brings rice chips from a drawer while a person interferes.

TL;DR PaLM-E feeds images, state vectors and 3D object tokens into the PaLM language model, which then writes robot plans as text. It is co-trained on robot and web vision-language data. With 320 examples per task, co-training lifts TAMP planning success from about 31% to 74%, and the 562B model scores 66.1 on OK-VQA.

What they do

LLM planners such as SayCan see only text, so they miss scene geometry. Off-the-shelf vision-language models cannot solve robot reasoning tasks either. PaLM-E encodes images, state vectors or object-centric scene tokens into PaLM’s word-embedding space and interleaves them with text as “multimodal sentences”. It outputs text: either an answer, or the next plan step for existing language-conditioned low-level policies to execute.

How it works

Encoders. An MLP for state vectors; ViT-4B or ViT-22B with a learned projection; per-object ViT tokens from ground-truth masks; or OSRT 3D object slots. Entity tokens (Object 1 is <obj_1>) let plans refer to specific objects.
Scale and training. PaLM 8B, 62B or 540B gives PaLM-E-12B, -84B and -562B. The LLM is either frozen, so only the encoders train, or finetuned end-to-end.
Full mixture. Training is mostly web vision-language data. Only 8.9% is robot data, drawn from simulated TAMP, Language-Table and kitchen mobile manipulation.
Closed loop. On Language-Table, PaLM-E sends subgoals at 1 Hz to policies running at 5 Hz. In the kitchen it picks RT-1 skills step by step until it outputs “terminate”, replanning from each new image.

Results

TAMP planning (320 examples per task, frozen LLM): ViT-4B succeeds 30.6% and 32.9% when trained on that robot alone, versus 74.1% and 74.6% with the full mixture. OSRT reaches 82.5% and 76.2%. SayCan with oracle affordances gets 38.7% and 33.3%.
Mobile manipulation (best PaLM-E-12B variant, F1 score): failure detection 0.91 (a CLIP-based detector built for this dataset: 0.89); affordance prediction 0.91 (QT-OPT: 0.63).
Vision-language benchmarks. One PaLM-E-562B checkpoint scores 66.1 on OK-VQA, above the task-finetuned PaLI (64.5).
Language forgetting. End-to-end training costs PaLM-E-12B 87.3% of its relative language-generation performance, but PaLM-E-562B only 3.9%.

Limitations

PaLM-E only sequences a small vocabulary of existing low-level skills; it does not output motor actions. Real-robot long-horizon planning is shown only qualitatively. A frozen LLM sometimes struggled on robot tasks, and finetuning the smaller models erased much of their language ability.

Videos

PaLM-E issues subgoals that guide a tabletop robot sorting blocks into corners.

Long-horizon block sorting on the real tabletop robot while a person moves blocks.

Figures

From Figure 1: One multimodal language model for robot planning, visual question answering and language tasks. Images... — From Figure 1: One multimodal language model for robot planning, visual question answering and language tasks. Images, neural 3D representations or states are inserted among the text tokens as input to an LLM trained end-to-end.

From Figure 3: In three robot domains, PaLM and ViT pretraining plus the full mixture of robot and vision-language da... — From Figure 3: In three robot domains, PaLM and ViT pretraining plus the full mixture of robot and vision-language data beats models trained only on in-domain data.

From Figure 5: One PaLM-E model directs the low-level policies of two real robots: a long-horizon mobile manipulation... — From Figure 5: One PaLM-E model directs the low-level policies of two real robots: a long-horizon mobile manipulation task in a kitchen, and one-shot and zero-shot tabletop tasks.

Abstract

Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model, PaLM-E-562B with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong