TidyBot: Personalized Robot Assistance with Large Language Models
TL;DR A user gives a few example object placements, an LLM summarizes them into general rules, and a mobile manipulator uses those rules to tidy a room. The approach reaches 91.2% accuracy on unseen objects in a text benchmark, and the real robot puts away 85.0% of objects.
What they do
Where things belong in a home depends on the person. TidyBot learns one user’s preferences from a handful of example placements, without collecting a preference dataset or training a model. An off-the-shelf LLM summarizes the examples into general rules (“light-colored clothes go in the drawer, dark-colored clothes in the closet”), and a mobile manipulator follows those rules to clear objects off the floor.
How it works
- Summarize, then apply. Examples are written as Pythonic
pick_and_place(object, receptacle)calls. The LLM (text-davinci-003) completes a# Summary:comment, and that summary is then used to place unseen objects. The same scheme picks a manipulation primitive per object (pick-and-place or pick-and-toss). - Categories for perception. Nouns extracted from the summary (e.g. “clothing”, “fruit”) become CLIP’s label set, so the classifier only has to separate 2-5 general categories.
- Robot. A holonomic base with a Kinova Gen3 arm. Overhead cameras track the base and find objects with ViLD, and receptacle locations are hard-coded. The robot repeatedly drives to the closest object, classifies a close-up image, and places or tosses it into its receptacle.
Results
- On a 96-scenario text benchmark, summarization gets 91.2% on unseen objects, against 78.5% when the LLM sees only the examples, 83.7% for CLIP embeddings and 67.5% for WordNet. Human-written summaries reach 97.5%.
- In a 40-person user study, placements from summarization were preferred 46.9% of the time versus 19.1% for CLIP embeddings.
- Real robot, 8 scenarios × 3 runs (240 objects): 85.0% put away correctly. Localization found 92.5% of objects and CLIP classified 95.5% of those. Each object took 15-20 s.
Limitations
Summaries sometimes just list the seen objects or merge receptacles (top and bottom drawer into “drawers”). The robot uses hand-written primitives, top-down grasps and known receptacle locations, and cannot drive over objects, so heavy clutter is a problem.
Videos
Figures
Abstract
For a robot to personalize physical assistance effectively, it must learn user preferences that can be generally reapplied to future scenarios. In this work, we investigate personalization of household cleanup with robots that can tidy up rooms by picking up objects and putting them away. A key challenge is determining the proper place to put each object, as people's preferences can vary greatly depending on personal taste or cultural background. For instance, one person may prefer storing shirts in the drawer, while another may prefer them on the shelf. We aim to build systems that can learn such preferences from just a handful of examples via prior interactions with a particular person. We show that robots can combine language-based planning and perception with the few-shot summarization capabilities of large language models (LLMs) to infer generalized user preferences that are broadly applicable to future interactions. This approach enables fast adaptation and achieves 91.2% accuracy on unseen objects in our benchmark dataset. We also demonstrate our approach on a real-world mobile manipulator called TidyBot, which successfully puts away 85.0% of objects in real-world test scenarios.
Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.