| CARVIEW |
Perspective-Aware Reasoning in Vision-Language Models
via Mental Imagery Simulation
📝 TL;DR: With our framework, VLMs can perform spatial reasoning in arbitrary perspectives.
Abstract
We present a framework for perspective-aware reasoning in vision-language models (VLMs) through mental imagery simulation. Perspective-taking—the ability to perceive an environment or situation from an alternative viewpoint—is a key benchmark for human-level visual understanding, essential for environmental interaction and collaboration with autonomous agents. Despite advancements in spatial reasoning within VLMs, recent research has shown that modern VLMs significantly lack perspective-aware reasoning capabilities and exhibit a strong bias toward egocentric interpretations. To bridge the gap between VLMs and human perception, we focus on the role of mental imagery, where humans perceive the world through abstracted representations that facilitate perspective shifts. Motivated by this, we propose a framework for perspective-aware reasoning, named Abstract Perspective Change (APC), that effectively leverages vision foundation models, such as object detection, segmentation, and orientation estimation, to construct scene abstractions and enable perspective transformations. Our experiments on synthetic and real-image benchmarks, compared with various VLMs, demonstrate significant improvements in perspective-aware reasoning with our framework, further outperforming fine-tuned spatial reasoning models and novel-view-synthesis-based approaches.
👀 Problem Definition
Perspectives are important, but remain a missing block in VLM spatial reasoning.
🤔 How Do Humans Shift Perspectives?
We can imagine a scene from another viewpoint by forming an abstract version of it and rotating it in our mind.
- Given an image of a scene and a question based on it, how can we turn it into an abstract 3D scene?
- After transforming the abstract scene to align with a given perspective, how can we feed it back to the VLM?
💡 Abstract Perspective Change
We propose APC, a framework that enables VLMs to perform perspective-aware reasoning,
by explicitly simulating the mental imagery process of humans.
- Scene Abstraction (Sec. 3.1): APC first detects the objects of interest and build a coarse 3D abstraction of the scene using off-the-shelf vision foundation models. We employ models for object detection [12], segmentation [13], depth estimation [14], and orientation estimation [15].
- Perspective Change (Sec. 3.2): Then, a reference perspective is set and the abstraction is transformed into the reference viewer’s egocentric coordinate frame.
- Perspective Prompting (Sec. 3.3): Finally, APC passes the transformed scene to the VLM by producing (1) a numerical (textual) prompt or (2) an abstract visual prompt, and poses the question of interest from the reference perspective.
🔍 Perspective Prompts
In APC, the transformed abstract scene is then passed to the VLM in the form of a prompt.
🖼️ Results
Comparison with Baseline VLMs.
Spatial Reasoning with Perspective Change. Recent VLMs such as Qwen2.5-VL [10] and Cambrian-1 [11] often struggle with spatial reasoning tasks that require a shift to a specific reference viewpoint. In constrast, our APC effectively handles such perspective changes by constructing a scene abstraction and delivering the transformed view through a simple prompting technique.
Probing the Perspective Awareness of VLMs.
Perspective Awareness. Each plot shows accuracy versus the angular offset θ between the camera and the reference viewpoint. While baselines show clear degradation at certain ranges of θ, APC retains robust accuracy across all angles, demonstrating strong perspective-aware reasoning.
References
[1] Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities, Zhang et al., ICLR 2025[2] 3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark, Ma et al., arXiv 2024
[3] Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Models, Góral et al., arXiv 2024
[4] The 3D-PC: A Benchmark for Visual Perspective Taking in Humans and Machines, Linsley et al., ICLR 2025
[5] Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces, Yang et al., CVPR 2025
[6] Principles of mental imagery., Finke, R. A, The MIT Press, 1989
[7] Mental imagery., Bence Nanay, The Stanford Encyclopedia of Philosophy, 1997
[8] Mental rotation of three-dimensional objects., Roger N Shepard and Jacqueline Metzler, Science, 1971
[9] Visual images preserve metric spatial information: Evidence from studies of image scanning., Kosslyn et al., Journal of Experimental Psychology, 1978
[10] Qwen2.5-VL Technical Report, Qwen Team Alibaba Group, arXiv 2025
[11] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs, NeurIPS 2024
[12] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection, Liu et al., ECCV 2024
[13] Segment Anything, Kirillov et al., ICCV 2023
[14] Depth Pro: Sharp Monocular Metric Depth in Less Than a Second, Bochkovskii et al., ICLR 2025
[15] Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models, Wang et al., arXiv 2024
Citation
If you find our work helpful, please cite the following paper.
@inproceedings{lee2025perspective,
title = {Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulation},
author = {Lee, Phillip Y. and Je, Jihyeon and Park, Chanho and Uy, Mikaela Angelina and Guibas, Leonidas and Sung, Minhyuk},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision},
year = {2025}
}