Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Code as Policies: Language Model Programs for Embodied Control

Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, Andy Zeng

Seminal September 2022

Overview montage: tabletop rearrangement, whiteboard drawing and a mobile robot following language commands.

TL;DR Few-shot prompt a code-writing LLM to turn language commands into Python robot policies that call perception and control APIs, with no training. On unseen simulated tabletop tasks it succeeds 80% of the time on long-horizon instructions, where CLIPort trained on 30k demonstrations gets 0%.

What they do

In Code as Policies (CaP), a code-writing LLM writes the robot policy itself rather than choosing from a fixed set of learned skills. Prompted with example commands (as Python comments) and their code, Codex (code-davinci-002) writes new Python for unseen commands. That code reads perception outputs, does geometry with NumPy or Shapely, loops on feedback and calls control primitives. Nothing is trained.

How it works

  • Prompts. Each language model program (LMP) gets Hints (import lines naming the available APIs) and Examples (instruction-to-code pairs). A session keeps appending commands and responses, so later commands can say “undo that”.
  • APIs as grounding. Perception comes from off-the-shelf open-vocabulary detectors (MDETR, ViLD) through calls like get_obj_pos. Actions are scripted primitives such as put_first_on_second or trajectory following.
  • Hierarchical code-gen. High-level LMPs call other LMPs (e.g. parse_obj for “the left most block”). Functions that are used but not yet defined are found by parsing the syntax tree, then written recursively by a function-generation LMP.

Results

  • RoboCodeGen, their new benchmark of 37 robotics coding problems: hierarchical code-gen passes 95% with code-davinci, vs 81% flat. Bigger models do better: 5% for 6.7B GPT-3, 84% for 175B GPT-3.
  • HumanEval: 39.8% P@1 and 53.0% greedy, vs 34.9% and 45.7% for flat generation.
  • Simulated tabletop, 50 trials per task, unseen attributes and instructions: CaP gets 80% on long-horizon tasks and 62% on spatial-geometric ones. CLIPort scores near zero on both, and a natural-language LLM planner gets 64% on long-horizon. With seen attributes and instructions, CLIPort beats CaP on spatial-geometric tasks (97.3% vs 89.3%).
  • On spatial selection questions, code gets 98% accuracy vs 58% for chain-of-thought in natural language.

Limitations

CaP can only use what the perception APIs can describe and the control primitives that exist, and only a few primitive parameters fit in a prompt. It struggles with commands much longer or more abstract than the examples (e.g. “build a house with the blocks”). It assumes every instruction is feasible and cannot tell in advance whether its code is correct. Real-robot demos are qualitative only, and many real-world failures came from inaccurate detections.

Videos

A UR5e with a suction gripper rearranging blocks around bowls from spoken commands.
The UR5e drawing a hexagon on a blank whiteboard from a language command.

Figures

From Figure 1: Given few-shot examples, a code-writing LLM translates a command (
From Figure 1: Given few-shot examples, a code-writing LLM translates a command ("Stack the blocks on the empty bowl") into policy code that calls perception and control APIs and defines its own missing functions.
From Figure 2: Natural language instructions followed on table-top manipulation (a, b), 2D shape drawing (c) and mobi...
From Figure 2: Natural language instructions followed on table-top manipulation (a, b), 2D shape drawing (c) and mobile manipulation in a kitchen with an Everyday Robots robot (d).
Abstract

Large language models (LLMs) trained on code completion have been shown to be capable of synthesizing simple Python programs from docstrings [1]. We find that these code-writing LLMs can be re-purposed to write robot policy code, given natural language commands. Specifically, policy code can express functions or feedback loops that process perception outputs (e.g.,from object detectors [2], [3]) and parameterize control primitive APIs. When provided as input several example language commands (formatted as comments) followed by corresponding policy code (via few-shot prompting), LLMs can take in new commands and autonomously re-compose API calls to generate new policy code respectively. By chaining classic logic structures and referencing third-party libraries (e.g., NumPy, Shapely) to perform arithmetic, LLMs used in this way can write robot policies that (i) exhibit spatial-geometric reasoning, (ii) generalize to new instructions, and (iii) prescribe precise values (e.g., velocities) to ambiguous descriptions ("faster") depending on context (i.e., behavioral commonsense). This paper presents code as policies: a robot-centric formulation of language model generated programs (LMPs) that can represent reactive policies (e.g., impedance controllers), as well as waypoint-based policies (vision-based pick and place, trajectory-based control), demonstrated across multiple real robot platforms. Central to our approach is prompting hierarchical code-gen (recursively defining undefined functions), which can write more complex code and also improves state-of-the-art to solve 39.8% of problems on the HumanEval [1] benchmark. Code and videos are available at https://code-as-policies.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.