• ArXiv (paper): NavSpace — How Navigation Agents Follow Spatial Intelligence Instructions
• Project website: https://navspace.github.io/

Visit the paper for full technical details and the project website for demos, visualizations, and additional resources.

Instruction-following navigation is a key step toward embodied intelligence. Prior benchmarks mainly focus on semantic understanding but overlook systematically evaluating navigation agents' spatial perception and reasoning capabilities. In this work, we introduce the NavSpace benchmark, which contains six task categories and 1,228 trajectory–instruction pairs designed to probe the spatial intelligence of navigation agents. On this benchmark, we comprehensively evaluate 22 navigation agents, including state-of-the-art navigation models and multimodal large language models. The evaluation results lift the veil on spatial intelligence in embodied navigation. Furthermore, we propose SNav, a new spatially intelligent navigation model. SNav outperforms existing navigation agents on NavSpace and real robot tests, establishing a strong baseline for future work.

The NavSpace dataset contains vision-and-language navigation data for 6 sub-tasks, totaling 1,228 episodes.

Each subfolder contains three different JSON file formats:

1. *_vln.json:

   • Standard format used by Vision-and-Language Navigation (VLN) models
   • Contains complete episode information: coordinates, instructions, paths, etc.
   • Serves as the baseline file for data consistency validation

2. *_action.json:

   • Contains ground-truth (GT) action sequences
   • Fully consistent with vln.json in core fields
   • Additional detailed action sequences and reference paths included

3. *_with_tokens.json:

   • Pre-tokenized format designed for lightweight navigation models
   • Provides pre-tokenized instruction text for efficient processing
   • Scene paths may differ from other files (due to different training environments)

• forward: Move 0.25 meters straight
• left/right: Rotate 30° left or right
• look-up/look-down: Tilt camera up or down by 30°
• backward: Move 0.25 meters backward
• stop: End of trajectory

• All files maintain consistency in core fields: episode_id, instruction_text, coordinates, goal points, etc.
• scene_id paths may vary due to different environments, but scene filenames remain the same
• Data integrity ensured through validation program

1. Use validate_dataset_integrity.py to verify data integrity
2. Use *_vln.json as baseline for data comparison
3. Choose appropriate data format based on model requirements

• Evaluation program is available as gpt_eval.py, which requires a GPT API key.
• To run the evaluation process, execute the bash script el.sh
• Additional evaluation methods will be released soon...