To assess whether VLMs can understand the physical world, we evaluated 75 representative VLMs on PhysBench and found that:

• VLMs exhibit limited understanding of the physical world.
• Closed-source models generally perform better.

To assess whether VLMs can understand the physical world, we evaluated 75 representative VLMs on PhysBench and found that:

• Physical world understanding differs significantly from common VQA tasks. We established a correlation map with the common VQA benchmark. Our analysis identifies a notable distinction between PhysBench and traditional VLM benchmarks, with PhysBench demonstrating a closer alignment with MMMU, which necessitates complex reasoning and diverging from the majority of other benchmarks.
• VLMs's physical world understanding ability does not scale with model size, data, or frames.While keeping the data size constant, increasing the model size or, alternatively, increasing the data size while keeping the model size unchanged, led to inconsistent results. Similarly, increasing the number of frames also yielded unstable outcomes.

We were perplexed by the fact that increasing the amount of training data did not improve the VLM's understanding of the physical world. To investigate further, we examined the training datasets of LLaVA-1.5, VILA-1.5, and PLLaVA-1.5 and identified a lack of physical world knowledge in these datasets. Additionally, keywords frequently encountered in PhysBench are notably rare in the training data of these model. This deficiency in relevant data likely contributes to the VLM's poor comprehension of physical world concepts. We further support this hypothesis by analyzing the error distribution and fine-tuning the VLM in subsequent experiments.

To investigate the poor performance of VLMs on PhysBench, we randomly selected 500 questions and obtained explanations from three models—GPT-4o, Phi-3V, and Gemini-1.5-flash. Expert annotators classified the root causes of the mispredictions into six categories: perception errors, reasoning errors, lack of knowledge, refusal to answer, failure to follow instructions, and annotation errors in the dataset. We find that perceptual and knowledge gaps constitute the majority of errors.

Our error analysis revealed that inadequate physical world knowledge and reasoning capabilities were key contributors to the models’ poor performance. To investigate whether introducing additional examples could enhance performance, we conducted tests on 200 entries of PhysBench, pairing each with a similar example. These additional examples were incorporated through fine-tuning or in-context learning. As shown in the below figure, the performance improvements after adding physical world knowledge examples indicate that VLMs can transfer physical knowledge to some extent. This suggests that the original data’s lack of physical world knowledge was a significant factor in the models’ suboptimal performance.