Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners

Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, Anirudha Majumdar

Seminal July 2023

Told to put the bowl in the microwave, the robot asks "metal or plastic?" and then acts on the answer.

TL;DR KnowNo frames each step of an LLM planner as a multiple-choice question and uses conformal prediction to keep only the options it cannot rule out. When more than one option remains, the robot asks a human. It meets a user-chosen task success rate with 10-24% less human help than baselines.

What they do

LLM planners can be confidently wrong, and instructions are often ambiguous: “place the bowl in the microwave” when there is a plastic and a metal bowl. KnowNo asks for help only when needed. With human help, the robot succeeds at a user-chosen rate 1-ε while keeping the amount of help small. No fine-tuning.

How it works

Multiple-choice scoring. The LLM writes four different next steps, plus “an option not listed here”. The next-token probabilities of the labels A-E are the scores, which avoids the length bias of scoring whole sentences.
Conformal prediction (CP). 400 labeled calibration scenarios set a score threshold. Options above it form the prediction set. One option: the robot acts. More: a human picks one.
Multi-step. Calibration runs over whole sequences, each scored by its least-confident step. Per-step sets with the same threshold keep the guarantee, even though actions change later inputs.

Results

Simulated tabletop rearrangement (ε = 0.15) with attribute, numeric and spatial ambiguity: KnowNo strays least from the target success rate. In the more ambiguous settings it needs up to 24% less help than Simple Set (the same scores thresholded without CP).
UR5 food sorting by preference, 50 trials, plan success matched at 0.76: help on 0.58 of steps vs. 0.72. Task success is 0.74 vs. 0.72, and 0.38 with no help.
Kitchen mobile manipulator: help 0.67 vs. 0.81. Task success is 0.76 because low-level actions succeed only about 86% of the time.
Bimanual KUKA arms: 9 of 10 trials succeed.

Limitations

The guarantee assumes the text input describes the scene correctly, low-level actions succeed and the human answers truthfully. Labeling the multi-step calibration set took about 4 hours. GPT-3.5 favored options D and E, so it needed more help (0.86).

Videos

The mobile manipulator at the office-kitchen recycling, compost and landfill bins.

Bimanual KUKA setup, where each arm can reach only its own side of the table.

Figures

From Figure 1: Given an instruction, an LLM generates possible next steps and a confidence score for each. Conformal ... — From Figure 1: Given an instruction, an LLM generates possible next steps and a confidence score for each. Conformal prediction keeps the options that score above a calibrated quantile, and the robot asks for help if more than one remains.

From Figure 2: Planning is posed as multiple-choice Q&A. The LLM generates plausible options and is then asked which ... — From Figure 2: Planning is posed as multiple-choice Q&A. The LLM generates plausible options and is then asked which one is correct. Using next-token likelihoods on a calibration set, conformal prediction picks a threshold, and every test option that scores above it goes into the prediction set. The set covers the true option with the user-specified probability.

From Figure 4: Task success rate against average prediction set size (left) and against human help rate (right) in si... — From Figure 4: Task success rate against average prediction set size (left) and against human help rate (right) in simulation, averaged over the three ambiguity settings, as the target error level varies. 200 trials per method.

Abstract

Large language models (LLMs) exhibit a wide range of promising capabilities -- from step-by-step planning to commonsense reasoning -- that may provide utility for robots, but remain prone to confidently hallucinated predictions. In this work, we present KnowNo, which is a framework for measuring and aligning the uncertainty of LLM-based planners such that they know when they don't know and ask for help when needed. KnowNo builds on the theory of conformal prediction to provide statistical guarantees on task completion while minimizing human help in complex multi-step planning settings. Experiments across a variety of simulated and real robot setups that involve tasks with different modes of ambiguity (e.g., from spatial to numeric uncertainties, from human preferences to Winograd schemas) show that KnowNo performs favorably over modern baselines (which may involve ensembles or extensive prompt tuning) in terms of improving efficiency and autonomy, while providing formal assurances. KnowNo can be used with LLMs out of the box without model-finetuning, and suggests a promising lightweight approach to modeling uncertainty that can complement and scale with the growing capabilities of foundation models. Website: https://robot-help.github.io

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong