| CARVIEW |
SimWorld
An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds
🌍 Open-ended Realistic Simulation
🤖 Rich LLM/VLM Agent Interface
💡 Diverse Reasoning Scenarios
Key Features
🌍 Open-ended Realistic Simulation
Realistic physical and social simulation with open-ended, language-controllable world generation.
🤖 Rich LLM/VLM Agent Interface
Gym-like interface, multimodal observations, and grounded natural-language actions spanning multiple levels of abstraction.
💡 Diverse Reasoning Scenarios
Support for diverse long-horizon physical and social reasoning, enabling systematic agent training and evaluation.
Simulator Comparison
| Simulator | Open-ended Realistic Simulation | Rich LLM/VLM Agent Interface | Diverse Reasoning Scenarios | |||||
|---|---|---|---|---|---|---|---|---|
| Simulation Realism | Procedural Generation | Language Control | Open Vocabulary Action Space | High-level Control | Low-level Control | Social Reasoning | Physical Reasoning | |
| SimWorld | ★★★ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| AI2-THOR | ★★ | ✓ | - | - | - | ✓ | - | ✓ |
| Genesis | ★★★ | ✓ | - | - | - | ✓ | - | ✓ |
| VirtualCommunity | ★★ | ✓ | - | - | - | ✓ | ✓ | ✓ |
| Mindcraft | ★ | ✓ | - | - | ✓ | ✓ | ✓ | - |
| Minedojo | ★ | ✓ | - | - | - | ✓ | - | - |
| MetaUrban | ★★ | ✓ | - | - | - | ✓ | - | ✓ |
| EmbodiedCity | ★★★ | - | - | - | - | ✓ | - | - |
| CARLA | ★★★ | - | - | - | - | ✓ | - | ✓ |
| GRUtopia | ★★ | - | - | - | - | ✓ | - | ✓ |
| OmniGibson | ★★ | - | - | - | ✓ | ✓ | - | ✓ |
| Habitat 3.0 | ★★ | - | - | - | - | ✓ | - | ✓ |
| UnrealZoo | ★★★ | - | - | - | - | ✓ | - | ✓ |
Open-ended Realistic Simulation
Procedural Scene Generation
SimWorld’s procedural generation system uses a modular, extensible pipeline with three stages: road generation, building generation, and street-element generation, each adding more structural and visual detail.
Various Environments
SimWorld offers a broad spectrum of meticulously designed environments, enabling diverse world-building and scenario development.
Loading video...
Physical and Social Dynamics
SimWorld simulates realistic physical, environmental, and social dynamics that shape the behavior of agents and the world around them.
Loading video...
Physical laws (e.g., gravity, momentum)
Loading video...
Lighting, weather, time of day
Loading video...
Traffic System
Language-based World Editing
Beyond static and procedurally generated maps, SimWorld supports open-ended, language-based world editing, allowing users and agents to create, modify, and compose scenes on the fly with natural-language commands.
Loading video...
“Generate several buildings that can fill the current empty block.”
Loading video...
“Generate a motorcycle and put it in the middle of the road.”
Loading video...
“Replace the buildings to make the overall style more consistent.”
Rich LLM/VLM Agent Interface
SimWorld provides a comprehensive interface for LLM/VLM agents with rich observation modalities and diverse action capabilities, enabling agents to perceive and interact with the environment in a natural and intuitive manner.
Observation Space
The simulator provides diverse observations including visual sensors (RGB, depth, segmentation), scene graph and GPS information (global and local maps).
RGB
Depth
Segmentation
Scene Graph
Global Map
Local Map
Open-Vocabulary Action Space
SimWorld supports an open-vocabulary action space that accepts natural language commands, which are then decomposed by a built-in action planner into sequences of low-level primitive actions.
Driving vehicles in realistic traffic
Natural social interaction between agents
Human–robot collaboration in shared spaces
Picking up and delivering objects
Fine-grained object manipulation
Pointing and gesturing to ground language
Diverse Reasoning Scenarios
Enable agents to perform complex reasoning and coordinated behaviors across diverse physical and social contexts.
Loading video...
Low-level motion control while avoiding obstacles.
Loading video...
Multimodal instruction-following navigation with visual hints.
Loading video...
Deliver food across the city, completing orders to earn money.
Research in SimWorld
SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds
Authors: Jiawei Ren*, Yan Zhuang*, Xiaokang Ye*, Lingjun Mao, Xuhong He, Jianzhi Shen, Mrinaal Dogra, Yiming Liang, Ruixuan Zhang, Tianai Yue, Yiqing Yang, Eric Liu, Ryan Wu, Kevin Benavente, Rajiv Mandya Nagaraju, Muhammad Faayez, Xiyan Zhang, Dhruv Vivek Sharma, Xianrui Zhong, Ziqiao Ma, Tianmin Shu†, Zhiting Hu†, Lianhui Qin†
Paper
DeliveryBench: Can Agents Earn Profit in Real World?
Authors: Lingjun Mao, Jiawei Ren, Kun Zhou, Jixuan Chen, Ziqiao Ma, Lianhui Qin
Coming soon!
SimWorld: An Open-ended Simulator for Agents in Physical and Social Worlds
Authors: Xiaokang Ye*, Jiawei Ren*, Yan Zhuang, Xuhong He, Yiming Liang, Yiqing Yang, Xianrui Zhong, Mrinaal Dogra, Eric Liu, Kevin Benavente, Rajiv Mandya Nagaraju, Dhruv Vivek Sharma, Ziqiao Ma, Tianmin Shu†, Zhiting Hu†, Lianhui Qin†
Venue: NeurIPS 2025 (Spotlight 🏆)
Paper
SimWorld-Robotics: Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration
Authors: Yan Zhuang, Jiawei Ren*, Xiaokang Ye*, Jianzhi Shen, Ruixuan Zhang, Tianai Yue, Muhammad Faayez, Xuhong He, Ziqiao Ma, Lianhui Qin†, Zhiting Hu†, Tianmin Shu†
Venue: NeurIPS 2025
Paper Repo Website
SimWorld: A World Simulator for Scaling Photorealistic Multi-Agent Interactions
Authors: Yan Zhuang*, Jiawei Ren*, Xiaokang Ye*, Xuhong He, Zijun Gao, Ryan Wu, Mrinaal Dogra, Cassie Zhang, Kai Kim, Bertt Wolfinger, Ziqiao Ma, Tianmin Shu†, Zhiting Hu†, Lianhui Qin†
Venue: CVPR 2025 Demo
Organizations
© 2025 SimWorld Team. All rights reserved.