Simulation and Real Benchmarks

To comprehensively evaluate the performance of VLAs, in this work, we benchmark all models on a diverse set of benchmarks and robotic manipulation tasks in both simulation and the real world. We choose two well-known and widely used simulation benchmarks, CALVIN, SimplerEnv, and a real-world robot manipulation benchmark, ByteDance Robot Benchmark.

Categorization of VLAs

We categorize VLA structures based on two primary designs: 1) INPUT whether the history is observable (horizontal axis); 2) OUTPUT: whether action is continuous or discrete (vertical axis).

👉Why do we prefer VLAs?

🔍Question 1   Are VLAs a good choice for building generalist robot policies?

• 📌Finding 1   VLA is the best choice in our extensive experiments to build generalist robot policies, in terms of robustness, generalization, and data efficiency.

Performance on CALVIN shows the state-of-the-art performance of the best VLA (KosMos P.H.) built by RoboVLMs.

On SimplerEnv, our model achieves the highest average performance on both WidowX + Bridge and Google Robot environments, demonstrating the general effectiveness and robustness against different settings and diverse manipulation tasks. **Note that our current results are evaluated without early stop, i.e., our policy ends only when the policy encounters the maximum rollout steps. With early stop, the results can be even higher.**

We investigated the impact of vision-language pre-training on the generalization and data efficiently. Vision-language pre-training is essential for both of them since an aligned vision-language representation provides a robust foundation for visual understanding, enabling the policy to focus on learning manipulation skills.

🔍Question 2   How do VLAs from RoboVLMs perform in real-world scenarios?

• 📌Finding 2   The best VLA built on RoboVLMs appears strong effectiveness and robustness in real-robot tasks, especially under unseen settings.

The best VLA built on RoboVLMs achieve the best performance in all real-world evaluation setups, extremely on Simple and Unseen Background, demonstrating their effectiveness and generalization ability.

👉How should we formulate VLAs?

🔍Question 3   What is the best-performing VLA structure?

• 📌Finding 3   The VLA achieves its best performance when using multi-step historical observations as inputs and continuous actions as outputs. For integrating history with continuous action space, the policy head structure performs better.

We demonstrate the ablation study of different VLA formulations on CALVIN benchmark over the effect of action space, history integration, and history organizing format. The results show significant improvements when taking multi-step historical observations as inputs, continuous actions as outputs, and a policy head to organize history.

🔍Question 4   How do different formulations affect the generalization and data efficiency for VLAs?

• 📌Finding 4   Leveraging policy head for history fusion is the best in terms of generalization and data efficiency.

We empirically study and evaluate the generalization and data efficiency of various VLA formulations, aiming to provide practical insights for training high-performing VLAs. Specifically, we assess the generalization and data efficiency of different VLAs built with RoboVLMs by training models with different architectures and formulations on varying data scales using the CALVIN datasets. Our best model, based on the KosMos backbone and leveraging a policy head for history fusion, exhibits only a slight performance drop in zero-shot settings. In contrast, other formulations experience significant performance declines. This finding highlights that the model architecture significantly impacts generalization.

For data efficienty, we observe trends similar to those for generalization. Our best model consistently achieves the highest performance when training data is scaled down, with a notably slower performance decline compared to other formulations. Additionally, comparisons of encoder-decoder VLAs at different scales reveal that larger models tend to be more data efficient.

👉Which VLM backbone is better for VLAs?

🔍Question 5   Which type of VLMs is most suitable for constructing VLAs?

• 📌Finding 5   KosMos and Paligemma are distinctively better than other VLMs in terms of training VLAs. We hypothesize they benefit from sufficient vision-language pretrain.

We base our VLAs on a diverse selection of pre-trained large-scale vision-language backbones with varying architectures, training data scales, model sizes, and latent embeddings. Through all of our experiments, we found that KosMos and Paligemma demonstrate the distinctively better performance possibly benefitting from sufficient vision-language pre-training. But it is still unclear and an open problem how other factors affect the resulting VLA due to large diversity of VLM backbones including training data, architecture, model size, data scale, LLM backbones, training recipes, etc.

👉When Should We Leverage Cross-Embodiment Datasets?

🔍Question 6   What types of data from large-scale cross-embodiment datasets are the most beneficial for building VLAs and when should we use them?

• 📌Finding 6   (1) Co-training with in-domain data, even from different tasks, proves beneficial for model performance; (2) Cross-embodiment data is more effective when utilized during the pre-training stage.

We conduct a series of experiments to investigate different strategies for using external large-scale cross-embodiment datasets, Open-X Embodiment. Results below demonstrate that cross-embodiment pre-training offers benefits to improve robustness as well as few-shot performance. While co-training with cross-embodiment data does not have significant improvements compared to using only in-domain data.

Rollout Examples

SimplerEnv - WidowX+Bridge

SimplerEnv - Google Robot

Real Robot: ByteDance Robot Benchmark

Unseen Distractor

Unseen Background

Unseen Target Object

Novel Skill Description

Acknowledgments

We thank all the members of the robotics research team at ByteDance Research for their assistance in real-world data collection, setup design, robot maintenance, and experiments. The author Minghuan Liu is supported by the ByteDance Scholarship.