Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

ProgPrompt: Generating Situated Robot Task Plans using Large Language Models

Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, Animesh Garg

Seminal September 2022

Real Franka arm sorting the fruits onto the plate and the bottle into the box.

TL;DR Prompt an LLM with a Python-style header (the robot's actions as imports, the scene's objects as a list, a few example task programs) and have it write the new task plan as code with comments and precondition assertions. With GPT-3 it reaches 0.34 success rate on VirtualHome tasks where a prior LLM planner gets 0, and it runs on a real Franka arm for tabletop pick-and-place.

What they do

LLM task planners either score every possible next action, which scales badly, or generate free text that may name actions the robot lacks or objects that are not there. ProgPrompt writes the prompt as a Python program that spells out the robot’s actions and the scene’s objects, and the LLM completes a new function whose body is the plan.

How it works

  • Prompt. Action primitives appear as import statements, the available objects as a list of strings, then three example task programs and the header of the new task (e.g. def microwave_salmon():). Changing robots only means changing the imports.
  • Plan structure. Natural-language comments split the task into sub-tasks, and assert statements check preconditions with recovery actions, e.g. if the agent is not close to the salmon, find('salmon') before grab('salmon').
  • Execution. An interpreter runs each action against the environment. In VirtualHome, assertions are checked by giving the LLM the relevant part of the state graph and asking whether the assertion holds.
  • Real robot. A Franka Panda gets one primitive, grab_and_putin(obj1, obj2), backed by an existing pick-and-place policy; ViLD detects the objects for the object list. No assertions are used on the robot.

Results

  • VirtualHome, 10 test tasks x 5 runs: GPT-3 ProgPrompt gets 0.34 success rate, 0.84 executability and 0.65 goal-condition recall. The planner of Huang et al. gets 0.00 / 0.45 / 0.21; Codex does best at 0.40 / 0.90 / 0.72.
  • Removing both comments and assertions drops success to 0.18; removing only comments gives 0.30 and only assertions 0.28.
  • In two more VirtualHome scenes success is 0.56 in each. Tasks with 11-18 ground-truth steps never fully succeed (GCR 0.36).
  • On 4 tabletop tasks with up to 4 distractors, every plan was executable. Only the sort task failed twice: once from a gripper failure, once because the LLM took a soup can for a bottle.

Limitations

Generated assertions do not cover every case (e.g. not opening a cabinet that holds the object), there is no feedback on whether an action succeeded, and some plans are cut short by LLM API length caps. Evaluation against a single ground-truth final state counts valid alternative solutions as failures. The robot runs open-loop, and the authors call its results a qualitative demonstration.

Videos

VirtualHome agent executing a generated plan for "put salmon in the fridge".
VirtualHome agent executing a generated plan for "throw away apple".

Figures

From Figure 1: The prompt imports the robot's actions, lists the scene objects and gives example task functions. GPT-...
From Figure 1: The prompt imports the robot's actions, lists the scene objects and gives example task functions. GPT-3 completes a new function, here sorting fruits onto the plate and the bottle into the box, which the robot executes directly.
From Figure 2: The planning prompt (imports, object list, example tasks) and the generated plan for
From Figure 2: The planning prompt (imports, object list, example tasks) and the generated plan for "microwave salmon", with comments, action calls and assertions with recovery steps. A second prompt checks each assertion against the current state; the full execution in VirtualHome is shown bottom right.
From Figure 4: Real-robot rollout of the sorting task. The plan puts the banana and strawberry on the plate and the b...
From Figure 4: Real-robot rollout of the sorting task. The plan puts the banana and strawberry on the plate and the bottle in the box, and ignores the distractor drill.
Abstract

Task planning can require defining myriad domain knowledge about the world in which a robot needs to act. To ameliorate that effort, large language models (LLMs) can be used to score potential next actions during task planning, and even generate action sequences directly, given an instruction in natural language with no additional domain information. However, such methods either require enumerating all possible next steps for scoring, or generate free-form text that may contain actions not possible on a given robot in its current context. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments, robot capabilities, and tasks. Our key insight is to prompt the LLM with program-like specifications of the available actions and objects in an environment, as well as with example programs that can be executed. We make concrete recommendations about prompt structure and generation constraints through ablation experiments, demonstrate state of the art success rates in VirtualHome household tasks, and deploy our method on a physical robot arm for tabletop tasks. Website at progprompt.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.