Website | Paper | ArXiv

Bhargav Chandaka*, Gloria X. Wang*, Haozhe Chen, Henry Che, Albert J. Zhai, Shenlong Wang

University of Illinois Urbana-Champaign

CoRL 2025

• Nvidia GPU (Tested on 4070 and 4090)
• For robot hardware, look at our hardware design doc.

Our system uses ROS 2, which requires Linux either natively or through WSL 2.

We use two docker containers: one for the ROS 2 workspace and the other for Isaac Sim

• make docker: to build the docker container specified by Dockerfile
• make: to enter docker and builds packages under src/ using colcon build --symlink-install so changes won't require rebuilding
• make dev: to enter docker without colcon build
• make exit: to close the container

• install nvidia-container-toolkit
• make isaacsim: to start the Isaac Sim docker
• download Isaac Sim Streaming Client from the release page
• make the streaming client an executable chmod +x isaacsim-webrtc-streaming-client-1.0.6-linux-x64.AppImage
• launch the streaming client ./isaacsim-webrtc-streaming-client-1.0.6-linux-x64.AppImage

We use the OpenAI API key for VLM calls to GPT-4.1 and Bing Search API key for web search.

Add both keys in a .env file under src/mm_dev/config/ with the format:

OPENAI_API_KEY=<your_key_here>
BING_API_KEY=<your_key_here>

Build the NanoOWL model (only need to do this once).

make docker
mkdir data
python3 -m nanoowl.build_image_encoder_engine data/owl_image_encoder_patch32.engine

Our system runs on ROS 2, and the launch commands are provided below. Run them in the order listed.

• Navigation and perception
  • For real: ros2 launch mm_bringup nav_3d.launch.py
  • For Isaac Sim: ros2 launch mm_bringup nav_isaacsim_3d.launch.py
• NPC chat (for simulation only): ros2 launch gennav_sim_control npc_chat.launch.py
• ReasonNav: ros2 launch mm_dev vlm_nav.launch.py

To run Isaac Sim:

• start the Isaac Sim docker in one terminal and launch the streaming client in another
• in the docker container, run sh start_isaacsim_docker.sh
• in the streaming client GUI, press CONNECT. if the screen is blank, go to View -> Reload. you may have to wait for a couple of seconds and reload
• in the navigation bar, go to GenNav -> GenNav. this will open up our extension
• scroll to Scene Controls and press LOAD. you may have to reload a couple times
• press RESET
• under Run Scenario, press RUN
• now you can run the ROS 2 commands

• If running simulation, make sure you press RUN before launching any ROS 2 nodes.
• Also ensure that you are restarting all ROS 2 nodes after reseting simulation, otherwise you may run into Lookup would require extrapolation into the past errors.
• for target_frame does not exist, restart the nodes and wait until the lidar scans show up in RViz before running vlm_nav.

Reach out with any other questions or feedback!

Please cite our paper if you find this repo useful:

@inproceedings{chandaka2025reasonnav,
    author={Chandaka, Bhargav and Wang, Gloria and Chen, Haozhe and Che, Henry and Zhai, Albert and Wang, Shenlong},
    title={Human-like Navigation in a World Built for Humans}, 
    booktitle={Conference on Robot Learning},
    year={2025}
}