RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control to boost generalization and enable emergent semantic reasoning. We propose to co-fine-tune state-of-the-art vision-language models on both robotic trajectory data and Internet-scale vision-language tasks. We express actions as text tokens and incorporate them directly into the training set. We refer to such models as vision-language-action models (VLA) and instantiate RT-2. Evaluation over 6k trials shows emergent capabilities including novel object generalization, icon/symbol interpretation, and multi-stage semantic reasoning.