VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models

Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, Li Fei-Fei

Seminal July 2023

Teaser: the real robot carries out "open the top drawer, and watch out for that vase".

TL;DR GPT-4 writes code that calls open-vocabulary perception to build 3D voxel maps of where the gripper should go and what it should avoid, and a motion planner follows them in closed loop with no training. On five real-robot tasks it succeeds 88% of the time (70% under disturbances), against 24% (0%) for an LLM calling motion primitives.

What they do

LLM-based robot planners usually call a library of hand-made motion primitives. VoxPoser instead has GPT-4 write Python that composes 3D voxel value maps in the robot’s observation space: where to go, what to avoid, how to rotate, how fast to move and when to close the gripper. These maps are the objective for a motion planner, so trajectories come out zero-shot, with no training of the LLM or the perception models.

How it works

Perception. For each object or part the LLM asks about, OWL-ViT finds a box, Segment Anything makes a mask and XMem tracks it; with RGB-D this gives a point cloud.
Value maps. A planner prompt splits the instruction into sub-tasks. A composer prompt then calls one prompt per map type, each returning a 100x100x100 voxel grid built with NumPy and the perception API.
Planning. Greedy search finds collision-free end-effector positions on a weighted sum of affordance and avoidance maps; the other maps set rotation, speed and gripper at each waypoint. The plan is redone at 5 Hz.
Online learning. For contact-rich tasks, noisy copies of the zero-shot trajectories guide exploration while an MLP dynamics model is learned.

Results

Real Franka Panda, 5 tasks, 10 trials each: 88% success on static scenes and 70% with disturbances, against 24% and 0% for an LLM-plus-primitives baseline in the style of Code as Policies.
Simulation, 13 tasks with 2766 instructions: on unseen instructions and attributes, 65.0% on object interaction and 76.7% on spatial composition, against 17.5% and 25.0% for LLM plus primitives and 0% for a learned U-Net costmap.
Opening a door, window and fridge in simulation: zero-shot success is 3-18%. Using those trajectories as an exploration prior reaches 80-92% after under 3 minutes of online interaction; exploring without the prior runs past the 12-hour limit.

Limitations

VoxPoser depends on external perception, which limits tasks that need holistic visual reasoning or fine object geometry. Most real-world errors came from the detector, which is sensitive to initial object poses and weaker on object parts. Contact-rich tasks still need a learned dynamics model, the planner handles only the end effector rather than the whole arm, and the prompts are engineered by hand.

Videos

Taking bread out of the toaster and laying it on a wooden board, 8x speed.

Sweeping paper trash into the blue dustpan, 8x speed.

Figures

From Figure 1: Through code, an LLM and a VLM compose a 3D value map from an instruction, which drives zero-shot traj... — From Figure 1: Through code, an LLM and a VLM compose a 3D value map from an instruction, which drives zero-shot trajectories for a wide range of everyday tasks.

From Figure 2: From RGB-D views and an instruction, LLM-written code calls VLMs to compose 3D affordance and constrai... — From Figure 2: From RGB-D views and an instruction, LLM-written code calls VLMs to compose 3D affordance and constraint maps (a), which a motion planner uses as its objective (b).

From Figure 3: Composed value maps and real-world rollouts. In the top row the maps move an object or part to a targe... — From Figure 3: Composed value maps and real-world rollouts. In the top row the maps move an object or part to a target; in the bottom two rows they guide the end effector.

Abstract

Large language models (LLMs) are shown to possess a wealth of actionable knowledge that can be extracted for robot manipulation in the form of reasoning and planning. Despite the progress, most still rely on pre-defined motion primitives to carry out the physical interactions with the environment, which remains a major bottleneck. In this work, we aim to synthesize robot trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a large variety of manipulation tasks given an open-set of instructions and an open-set of objects. We achieve this by first observing that LLMs excel at inferring affordances and constraints given a free-form language instruction. More importantly, by leveraging their code-writing capabilities, they can interact with a vision-language model (VLM) to compose 3D value maps to ground the knowledge into the observation space of the agent. The composed value maps are then used in a model-based planning framework to zero-shot synthesize closed-loop robot trajectories with robustness to dynamic perturbations. We further demonstrate how the proposed framework can benefit from online experiences by efficiently learning a dynamics model for scenes that involve contact-rich interactions. We present a large-scale study of the proposed method in both simulated and real-robot environments, showcasing the ability to perform a large variety of everyday manipulation tasks specified in free-form natural language. Videos and code at https://voxposer.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong