Matt Strong

Matt Strong

CS PhD at Stanford University

← Paper Digest

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, Andy Zeng

★ Must read April 2022

Throwing away a spilled coke and fetching a sponge, with each skill's language and affordance scores shown, 10x speed.

TL;DR SayCan picks each next step for a mobile manipulator by multiplying a language model's score for every skill description with that skill's learned value function, so the step is both useful and feasible. With PaLM it correctly plans 84% and executes 74% of 101 kitchen instructions.

What they do

A language model can break “I spilled my drink, can you help?” into steps, but it does not know what the robot can do or what is in front of it. SayCan restricts the LLM to scoring a fixed set of skill descriptions and weights each score by that skill’s value function, so the chosen step is both useful for the instruction and feasible in the current state.

How it works

  • Say. PaLM 540B, prompted with 17 example dialogs, scores the text of each skill (e.g. “find a sponge”) as the next step. The chosen skill is appended to the response and the query repeats until “done”.
  • Can. A skill’s affordance is its value function under a sparse 0/1 success reward. Pick values come from MT-Opt RL trained in simulation with RetinaGAN; go-to affordances use the distance to known object locations; place is always 1.0.
  • Skills. Pick actions come from a BC-Z-style policy trained on 68,000 teleoperated demonstrations plus 12,000 successful autonomous episodes. Navigation and placing use classical planners. The robot is an Everyday Robots mobile manipulator with a 7-DoF arm.

Results

  • Across 101 instructions in 7 families, PaLM-SayCan plans 84% and executes 74% correctly in the mock kitchen, and 81% and 60% in the real office kitchen.
  • Without value functions, planning drops to 67%; a generative LLM projected onto the nearest skill gets 74%. Giving the instruction straight to the BC policy succeeds 0% of the time.
  • With FLAN instead of PaLM, planning is 70% and execution 61%.
  • 65% of errors are LLM failures and 35% affordance failures. Long-horizon tasks execute 47%, mostly due to the LLM stopping early. Added drawer skills plan 100% but execute 33% over 21 queries.

Limitations

SayCan inherits the biases of the LLM and struggles with negation and ambiguous references. The main bottleneck is the range and reliability of the skills, and navigation assumes object locations are known. Feedback comes only from value functions at the current step, so it cannot react when a skill fails despite reporting a high value.

Videos

Long-horizon request: throw away three items, then bring a sponge to wipe the table, 10x speed.
Chain-of-thought prompting: the robot reasons that a grapefruit soda is fruit-flavored and caffeine-free, then fetches it, 6x speed.
Full video (YouTube)

Figures

From Figure 1: Asked
From Figure 1: Asked "I spilled my drink, can you help?", ungrounded LLMs suggest things the robot cannot do. SayCan grounds the LLM with value functions of pretrained skills and returns a plan the robot can execute.
From Figure 5: SayCan multiplies the language model's probability that a skill is useful for the instruction by the v...
From Figure 5: SayCan multiplies the language model's probability that a skill is useful for the instruction by the value function's probability that the skill succeeds, and runs the best skill. The skill is appended to the response and the models are queried again until the chosen step is to terminate.
From Figure 8: PaLM-SayCan's decision making for a spilled-coke request. At each step the top combined language and a...
From Figure 8: PaLM-SayCan's decision making for a spilled-coke request. At each step the top combined language and affordance score chooses the correct skill.
From Figure 7: Rollout of
From Figure 7: Rollout of "I just worked out, can you bring me a drink and a snack to recover?", with the nine planned steps drawn on an overhead map of the kitchen.
Abstract

Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language. However, a significant weakness of language models is that they lack real-world experience, which makes it difficult to leverage them for decision making within a given embodiment. For example, asking a language model to describe how to clean a spill might result in a reasonable narrative, but it may not be applicable to a particular agent, such as a robot, that needs to perform this task in a particular environment. We propose to provide real-world grounding by means of pretrained skills, which are used to constrain the model to propose natural language actions that are both feasible and contextually appropriate. The robot can act as the language model's "hands and eyes," while the language model supplies high-level semantic knowledge about the task. We show how low-level skills can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally-extended instructions, while value functions associated with these skills provide the grounding necessary to connect this knowledge to a particular physical environment. We evaluate our method on a number of real-world robotic tasks, where we show the need for real-world grounding and that this approach is capable of completing long-horizon, abstract, natural language instructions on a mobile manipulator. The project's website and the video can be found at https://say-can.github.io/.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.