Inner Monologue: Embodied Reasoning through Planning with Language Models

Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, Brian Ichter

Seminal July 2022

Kitchen robot asked for a soda, with the LLM's actions and success-detector feedback shown alongside.

TL;DR A frozen LLM plans robot skills step by step while success detection, scene descriptions and human answers are fed back into its prompt as text, so it can retry and replan. On a real kitchen mobile manipulator this raises success from 30.8% (SayCan) to 60.4%.

What they do

LLM planners such as SayCan pick skills assuming each one succeeds. Inner Monologue closes the loop: after every skill, textual feedback from the environment is appended to the LLM’s prompt, so it can retry, replan or ask a question. The LLM is only few-shot prompted, never trained. Tested on a simulated tabletop, a real tabletop arm and a real kitchen mobile manipulator.

How it works

Planner. A frozen LLM (InstructGPT on the tabletops, PaLM 540B in the kitchen) picks the next skill from a library of pretrained, language-described policies.
Feedback as text. Success says whether the last skill worked; Object lists visible objects; Scene lists which LLM-inferred sub-goals are achieved; Human lets the LLM ask open-ended questions and read the answers.
Feedback sources. Simulator state and heuristics in sim; MDETR detections plus bounding-box heuristics on the real tabletop; in the kitchen, a CLIP-based success detector finetuned on robot data and human-provided object lists, with SayCan’s affordance value functions still grounding actions.

Results

Simulated tabletop with injected noise: Object + Scene feedback reaches 82% and 86% on two unseen bowl-matching tasks, where a multi-task CLIPort policy scores 0%.
Real tabletop (10 runs per task): open-loop planning succeeds 20% overall; Object + Success feedback succeeds 90% (100% on block stacking, 80% on sorting fruits from bottles).
Kitchen (120 evaluations): 60.4% overall vs 30.8% for SayCan. With adversarial disturbances, mobile manipulation goes from 0% to 75%.
Unprompted behaviors include following goal changes, proposing a lighter block after a hint that one is too heavy, and taking instructions in Chinese, though not consistently.

Limitations

Scene feedback comes from scripted systems or human observers in the sim and kitchen setups. Success-detector false negatives cause extra retries and false positives hide failures from the planner; the LLM sometimes ignores feedback and proposes skills on absent objects. Overall performance is capped by what the low-level skills can do.

Videos

Real tabletop arm stacking blocks and sorting objects, retrying after failed placements, 2x speed.

Kitchen robot replanning as a human changes the goal twice mid-task, 4x speed.

Figures

From Figure 1: Inner Monologue combines perception models (scene descriptors and success detectors) with pretrained l... — From Figure 1: Inner Monologue combines perception models (scene descriptors and success detectors) with pretrained language-conditioned skills so an LLM can replan long-horizon tasks, shown for (a) mobile manipulation and (b,c) simulated and real tabletop manipulation.

From Figure 2: The types of textual feedback. Success detection reports whether a skill worked, passive scene descrip... — From Figure 2: The types of textual feedback. Success detection reports whether a skill worked, passive scene description gives structured scene information at every step, and active scene description answers questions only when the LLM asks.

From Figure 3: The three instantiations, simulated tabletop (top), real tabletop (middle) and real kitchen mobile man... — From Figure 3: The three instantiations, simulated tabletop (top), real tabletop (middle) and real kitchen mobile manipulation (bottom), each with its own prompts and feedback models. In the kitchen, actions are also grounded by pretrained affordance functions.

From Figure 5: Continued adaptation to new instructions. A human changes the goal mid-task, then asks the robot to fi... — From Figure 5: Continued adaptation to new instructions. A human changes the goal mid-task, then asks the robot to finish the previous task, and the planner switches both times.

Abstract

Recent works have shown how the reasoning capabilities of Large Language Models (LLMs) can be applied to domains beyond natural language processing, such as planning and interaction for robots. These embodied problems require an agent to understand many semantic aspects of the world: the repertoire of skills available, how these skills influence the world, and how changes to the world map back to the language. LLMs planning in embodied environments need to consider not just what skills to do, but also how and when to do them - answers that change over time in response to the agent's own choices. In this work, we investigate to what extent LLMs used in such embodied contexts can reason over sources of feedback provided through natural language, without any additional training. We propose that by leveraging environment feedback, LLMs are able to form an inner monologue that allows them to more richly process and plan in robotic control scenarios. We investigate a variety of sources of feedback, such as success detection, scene description, and human interaction. We find that closed-loop language feedback significantly improves high-level instruction completion on three domains, including simulated and real table top rearrangement tasks and long-horizon mobile manipulation tasks in a kitchen environment in the real world.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong