With the rising capabilities of LLMs and VLMs, the past two years have seen a surge in research work using VLMs for navigation and manipulation. Fusing the capabilities of visual interpretation with natural language processing, these models are poised to redefine how robotic systems interact with both their environment and human counterparts. The relevance of this topic cannot be overstated; as the frontier of human-robot interaction expands, so does the necessity for robots to comprehend and operate within complex environments using naturalistic instructions. Our workshop will not only reflect the state-of-the-art advancements in this domain, by featuring a diverse set of speakers, from senior academics to researchers in early careers, from industry researchers to companies producing mobile manipulation platforms, from researchers who are enthusiastic about using VLMs for robotics to those who have reservations about it. We aim for this event to be a catalyst for originality and diversity at ICRA 2024. We believe that, amidst a sea of workshops, ours will provide unique perspectives that will push the boundaries of what's achievable in robot navigation and manipulation.

In this workshop, we plan to discuss:

• How can VLMs/LLMs enhance robotics navigation and manipulation?
• How to extract world knowledge from pre-trained VLMs/LLMs and apply them to navigation and manipulation?
• How to integrate VLMs/LLMs with robot components, such as perception, control, and planning? How to account for partial observability and uncertainty?
• Benchmarks and datasets to assess the generalization capabilities of VLMs/LLMs for navigation and manipulation.
• Capabilities and limitations of VLMs/LLMs for navigation and manipulation (e.g. in task planning, spatial understanding)
• New interaction modes between robots and humans enabled by VLMs/LLMs.

All accepted workshop papers: OpenReview. Please bring a poster. We will have double-sided A0 portrait poster boards.