CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks

Oier Mees, Lukas Hermann, Erick Rosete-Beas, Wolfram Burgard

Seminal December 2021

Policies following language instructions such as "open the drawer" in four CALVIN environments.

TL;DR CALVIN is an open-source simulated benchmark in which a Franka arm must follow chains of five natural-language instructions across four related tabletop environments, learning from 24 hours of unlabeled teleoperated play. A multi-context imitation learning baseline solves 53.9% of single tasks but completes all five instructions in a row in only 0.08% of chains.

What they do

CALVIN (Composing Actions from Language and Vision) is a simulated benchmark for long-horizon, language-conditioned manipulation. A single agent must carry out a series of free-form instructions in a row, such as “open the drawer … pick up the blue block … push the block into the drawer”, with 7-DOF continuous control from onboard sensors. The authors also implement the MCIL baseline, which had no public implementation.

How it works

Environments. Four PyBullet environments (A-D) share a Franka Panda, a desk with a drawer, sliding door, button, switch and three colored blocks, but differ in textures and in where the static elements sit. 34 tasks are detected automatically from changes in simulator state.
Sensors and actions. Static and gripper RGB-D cameras, a vision-based tactile sensor and proprioception; absolute or relative Cartesian actions or joint actions at 30 Hz.
Data. About 24 hours of teleoperated play (about 2.4M steps) from three untrained operators. Only 1% is labeled with language, using 389 instructions (about 11 per task) applied procedurally, with MiniLM embeddings provided.
Evaluation. Train on D, on all four, or zero-shot on A, B, C with test on D. MTLC runs 10 rollouts per task; LH-MTLC runs 1000 chains of five subtasks, unseen instructions and a neutral robot start.

Results

Static-camera MCIL trained and tested on D solves 53.9% of the 34 tasks. Adding the gripper camera, depth or tactile input leaves this about the same (46.1-54.2%).
On five-instruction chains the same model completes the first subtask in 48.9% of chains, two in 12.9%, and all five in 0.08%. It sometimes confuses the red and blue blocks.
Zero-shot (A, B, C to D), static-camera MTLC drops to 38.6%; 0.2% of chains get through two subtasks and none through three.

Limitations

Objects are unicolored primitive shapes, chosen to keep the problem tractable, and everything runs in simulation. The only baseline is MCIL with absolute actions and no image augmentation. The authors note that its performance depends heavily on the robot’s initial position.

Videos

Figures

From Figure 3: A CALVIN scene, with a Franka Emika Panda arm at a desk that has a drawer, a sliding door, a button, a... — From Figure 3: A CALVIN scene, with a Franka Emika Panda arm at a desk that has a drawer, a sliding door, a button, a switch and three colored blocks.

From Figure 3: RGB view from the gripper camera, one of the supported sensors alongside the static RGB-D camera, grip... — From Figure 3: RGB view from the gripper camera, one of the supported sensors alongside the static RGB-D camera, gripper depth, proprioception and vision-based tactile sensing.

From Figure 7: How often each of the 34 subtasks appears in the 1000 instruction chains of the long-horizon evaluatio... — From Figure 7: How often each of the 34 subtasks appears in the 1000 instruction chains of the long-horizon evaluation, in percent.

Abstract

General-purpose robots coexisting with humans in their environment must learn to relate human language to their perceptions and actions to be useful in a range of daily tasks. Moreover, they need to acquire a diverse repertoire of general-purpose skills that allow composing long-horizon tasks by following unconstrained language instructions. In this paper, we present CALVIN (Composing Actions from Language and Vision), an open-source simulated benchmark to learn long-horizon language-conditioned tasks. Our aim is to make it possible to develop agents that can solve many robotic manipulation tasks over a long horizon, from onboard sensors, and specified only via human language. CALVIN tasks are more complex in terms of sequence length, action space, and language than existing vision-and-language task datasets and supports flexible specification of sensor suites. We evaluate the agents in zero-shot to novel language instructions and to novel environments and objects. We show that a baseline model based on multi-context imitation learning performs poorly on CALVIN, suggesting that there is significant room for developing innovative agents that learn to relate human language to their world models with this benchmark.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong