Findings and Takeaways

What Is the Impact of Tasks with Different Difficulties?

• Multi-view geometry inference is inherently harder than pair-view inference.
• Directly predicting dense 3D scene representations is much more challenging than estimating individual 3D attributes like depth and camera poses.
• Metric-scale depth estimation remains a key challenge for GFMs.
• Joint prediction of multiple geometric attributes (e.g., pose, depth, matching) may underlie recent performance gains.

Takeaway 1: Current GFMs are promising but face significant challenges when learning from overly complex tasks. Recommendation: Carefully decomposing difficult tasks (e.g., jointly predicting geometry, pose, depth, and tracking) into simpler sub-problems can facilitate more effective learning, especially under limited 3D data.

Do GFMs Generalize Well on Different Data Domains?

• GFMs struggle to generalize in domains with extreme data scarcity.

Takeaway 2: Diverse, high-quality data is critical for strong generalization. To improve robustness in underrepresented domains, GFMs must be trained on data that covers broader distributions and metric-scale annotations.

Hints for Model Architecture Design, ViT or Diffusion? Strong 2D Feature Extractor?

• No single design, feed-forward ViT or diffusion, is universally superior.
• Stronger 2D foundation models can significantly enhance 3D GFMs.

Takeaway 3: No single backbone—feed -forward ViT or diffusion, dominates; architecture choice should align with task needs. Moreover, leveraging strong 2D feature extractors (e.g., DINO) substantially boosts 3D performance.

Are Current GFMs Ready for Real-Time Perception Systems?

• Despite progress, GFMs still lack the efficiency required for real-time 3D applications.

Takeaway 4: As GFMs scale to handle more views and complex tasks, efficiency becomes as critical as accuracy for enabling real-time 3D perception.

Citation

@article{cong2025e3dbench,
  title={E3D-Bench: An End-to-End Benchmark for 3D Geometric Foundation Models},
  author={Cong, Wenyan and Liang, Yiqing and Zhang, Yancheng and Yang, Ziyi and Wang, Yan and Ivanovic, Boris and Pavone, Marco and Chen, Chen and Wang, Zhangyang and Fan, Zhiwen},
  journal={arXiv preprint arXiv:2506.01933},
  year={2025}
}