OpenVLA: An Open-Source Vision-Language-Action Model

Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, Chelsea Finn

★ Must read June 2024

WidowX robot putting the yellow corn on the pink plate in the BridgeData V2 sink, 1.5x speed.

TL;DR An open-source 7B vision-language-action model that fine-tunes a Llama 2 VLM with fused DINOv2 and SigLIP features to output discretized robot actions, trained on 970k Open X-Embodiment trajectories. It beats the 55B RT-2-X by 16.5% absolute success across 29 tasks on two robots, and LoRA fine-tuning matches full fine-tuning while training 1.4% of the parameters.

What they do

Existing VLAs such as RT-2-X are closed, and their fine-tuning to new robots was unexplored. OpenVLA is a 7B open-source VLA trained on 970k Open X-Embodiment trajectories, evaluated out of the box on two robots and fine-tuned to two new Franka setups.

How it works

Backbone. The Prismatic-7B VLM: SigLIP and DinoV2 patch features concatenated channel-wise, a 2-layer MLP projector, and Llama 2 7B. The input is one 224x224 image plus a language instruction.
Actions as tokens. Each action dimension is split into 256 bins between the 1st and 99th percentile of the training actions. The bins overwrite the 256 least-used Llama tokens, and the loss is next-token cross-entropy on action tokens only.
Training. OpenX single-arm datasets weighted with Octo’s mixture; 27 epochs with the vision encoder unfrozen, on 64 A100s for 14 days.

Results

BridgeData V2 WidowX (17 tasks, 170 rollouts): 70.6% vs 50.6% for RT-2-X, 20.0% for Octo and 18.5% for RT-1-X. On the Google robot (12 tasks) it gets 85.0%, comparable to RT-2-X’s 78.3%. RT-2-X is better at semantic generalization.
Fine-tuned on 10-150 demos, it averages 67.2% on Franka-Tabletop vs 48.5% for Diffusion Policy and 43.4% for Octo, and 58.3% on Franka-DROID vs 35.0% and 38.3%. Diffusion Policy still wins on narrow single-instruction tasks such as pouring corn.
LoRA (rank 32) reaches 68.2% vs 69.7% for full fine-tuning while training 1.4% of the parameters, on one A100 in 10-15 h.
4-bit inference matches bfloat16 (71.9% vs 71.3%) with less than half the memory. 8-bit drops to 58.1% because it runs at only 1.2 Hz on an A5000.

Limitations

Only a single image is used as input, with no proprioception or history. At about 6 Hz on an RTX 4090 in bfloat16, inference is too slow for 50 Hz setups like ALOHA. Success is typically below 90%. Base VLM size, co-training with web data and the choice of visual features were left unexplored.

Videos

Google robot picking up a Coke can.

Franka arm pouring corn from a bowl into a pot after fine-tuning, 7x speed.

Figures

From Figure 1: OpenVLA, a 7B-parameter open-source VLA trained on 970k robot episodes from Open X-Embodiment. It cont... — From Figure 1: OpenVLA, a 7B-parameter open-source VLA trained on 970k robot episodes from Open X-Embodiment. It controls multiple robots out of the box and can be adapted to new robots with parameter-efficient fine-tuning.

From Figure 2: OpenVLA architecture. An image and a language instruction pass through a vision encoder that concatena... — From Figure 2: OpenVLA architecture. An image and a language instruction pass through a vision encoder that concatenates DinoV2 and SigLIP features, a projector into the language embedding space, and a Llama 2 7B backbone that predicts 7-dimensional robot actions.

From Figure 3: BridgeData V2 WidowX tasks and success rates for visual, motion, physical and semantic generalization ... — From Figure 3: BridgeData V2 WidowX tasks and success rates for visual, motion, physical and semantic generalization and language grounding. OpenVLA has the highest overall success and beats RT-2-X in every category except semantic generalization, over 170 rollouts per method.

Abstract

Large policies pretrained on a combination of Internet-scale vision-language data and diverse robot demonstrations have the potential to change how we teach robots new skills: rather than training new behaviors from scratch, we can fine-tune such vision-language-action (VLA) models to obtain robust, generalizable policies for visuomotor control. Yet, widespread adoption of VLAs for robotics has been challenging as 1) existing VLAs are largely closed and inaccessible to the public, and 2) prior work fails to explore methods for efficiently fine-tuning VLAs for new tasks, a key component for adoption. Addressing these challenges, we introduce OpenVLA, a 7B-parameter open-source VLA trained on a diverse collection of 970k real-world robot demonstrations. OpenVLA builds on a Llama 2 language model combined with a visual encoder that fuses pretrained features from DINOv2 and SigLIP. As a product of the added data diversity and new model components, OpenVLA demonstrates strong results for generalist manipulation, outperforming closed models such as RT-2-X (55B) by 16.5% in absolute task success rate across 29 tasks and multiple robot embodiments, with 7x fewer parameters. We further show that we can effectively fine-tune OpenVLA for new settings, with especially strong generalization results in multi-task environments involving multiple objects and strong language grounding abilities, and outperform expressive from-scratch imitation learning methods such as Diffusion Policy by 20.4%. We also explore compute efficiency; as a separate contribution, we show that OpenVLA can be fine-tuned on consumer GPUs via modern low-rank adaptation methods and served efficiently via quantization without a hit to downstream success rate. Finally, we release model checkpoints, fine-tuning notebooks, and our PyTorch codebase with built-in support for training VLAs at scale on Open X-Embodiment datasets.

Summaries are written automatically from each paper's text. Figures and videos come from the paper and its project page; all credit goes to the authors.

Matt Strong