| CARVIEW |
Select Language
HTTP/2 200
server: GitHub.com
content-type: text/html; charset=utf-8
last-modified: Thu, 06 Nov 2025 02:43:29 GMT
access-control-allow-origin: *
etag: W/"690c0b51-11ca2"
expires: Sun, 28 Dec 2025 12:10:16 GMT
cache-control: max-age=600
content-encoding: gzip
x-proxy-cache: MISS
x-github-request-id: 3270:444BC:79CA51:8873EF:69511BD0
accept-ranges: bytes
age: 0
date: Sun, 28 Dec 2025 12:00:16 GMT
via: 1.1 varnish
x-served-by: cache-bom-vanm7210051-BOM
x-cache: MISS
x-cache-hits: 0
x-timer: S1766923217.587371,VS0,VE212
vary: Accept-Encoding
x-fastly-request-id: 1b155f30acbab2f81703a6c4d04592425e8e53d0
content-length: 12713
Xiaojian Ma's Homepage
Welcome to my academic homepage. I am a machine learning researcher. I received my Ph.D. in Computer Science at University of California, Los Angeles (UCLA) . I went to Tsinghua University for undergraduate study in Computer Science.
I work on multimodal learning for understanding, reasoning and skill learning. In particular, I'm interested in building models/agents that can learn from 2D/3D vision and text data, and perform a wide range of reasoning and embodied control tasks. Some of my research keywords can be found below:
- Multimodal learning: Vision and language, Visual reasoning, 3D vision, Generalist models
- Representation learning: Zero-shot and few-shot learning, Generative model
- Embodied agents: Reinforcement learning and imitation, Robotics, Sensor fusion
Email: jeasinema [at] gmail [dot] com / Google Scholar / LinkedIn
News
- [06/2025] Some of our recent work on robotic manipulation: Falcon (2-8x accelerated diffusion policy) , AdaReP (MPC w/ 10x efficiency) , ManiGen (synthetic manipulation data pump) , COIN (benchmarking VLAs on both dexterity and reasoning) ,
- [01/2025] Some recent efforts on perception for embodied agents (long-form videos, dynamic scenes): Embodied VideoAgent , LongViTU
- [12/2024] Some recent efforts on multi-modal, open-world agents: ROCKET-1 , Multi-modal Agent Tuning
- [10/2024] We will be hosting NeurIPS 2024 Workshop on Open-World Agents . Come join us in Vancouver, BC, Canada this winter!
Selected Publications
Preprint
Rujie Wu*,
Xiaojian Ma*,
Hai Ci,
Yue Fan,
Rongpeng Su,
Yuxuan Wang,
Haozhe Zhao*,
Qing Li,
Yizhou Wang,
LongViTU: Instruction Tuning for Long-Form Video Understanding
arXiv preprint / arXiv / Project
Synthetic data for long-form video understanding.
LongViTU: Instruction Tuning for Long-Form Video Understanding
arXiv preprint / arXiv / Project
Synthetic data for long-form video understanding.
Jun Guo*,
Xiaojian Ma*,
Yue Fan,
Huaping Liu,
Qing Li
Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting
arXiv preprint / arXiv / Project
Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting
arXiv preprint / arXiv / Project
Team CraftJarvis 
🐭 RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation
arXiv preprint / arXiv / Project / Demo
Significant improvement on creativity writing, codegen, and math problems.
🐭 RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation
arXiv preprint / arXiv / Project / Demo
Significant improvement on creativity writing, codegen, and math problems.
Sirui Xie,
Xiaojian Ma,
Peiyu Yu,
Yixin Zhu,
Ying Nian Wu and
Song-Chun Zhu
HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving
arXiv preprint / Paper / Click Here to Play HALMA! / arXiv
HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving
arXiv preprint / Paper / Click Here to Play HALMA! / arXiv
Conference and Journal
Huimin Wu*,
Xiaojian Ma*,
Haozhe Zhao*,
Yanpeng Zhao,
Qing Li
NEP: Autoregressive Image Editing via Next Editing Token Prediction
NeurIPS 2025 / arXiv / Project
Autoregressive T2I with native image editing and test-time scaling.
NEP: Autoregressive Image Editing via Next Editing Token Prediction
NeurIPS 2025 / arXiv / Project
Autoregressive T2I with native image editing and test-time scaling.
Yue Fan*,
Xiaojian Ma*,
Rongpeng Su,
Jun Guo,
Rujie Wu,
Xi Chen,
Qing Li
Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
ICCV 2025 / arXiv / Project
A memory framework for embodied agents in household environments (and beyond!)
Spotlight presentation.
Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
ICCV 2025 / arXiv / Project
A memory framework for embodied agents in household environments (and beyond!)
Spotlight presentation.
Team CraftJarvis
ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting
CVPR 2025 / arXiv / Project
Learning to interact with all your surroundings, from self-supervision, enables open-world agents :)
ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting
CVPR 2025 / arXiv / Project
Learning to interact with all your surroundings, from self-supervision, enables open-world agents :)
Zhi Gao*,
Bofei Zhang*,
Pengxiang Li*,
Xiaojian Ma,
Yue Fan,
Tao Yuan,
Yuwei Wu,
Yunde Jia,
Song-Chun Zhu,
Qing Li,
Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage
ICLR 2025 / arXiv / Project
Spotlight presentation.
Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage
ICLR 2025 / arXiv / Project
Spotlight presentation.
Team CraftJarvis
JARVIS-1: Open-world Multi-task Agents with Memory-Augmented Multimodal Language Models
T-PAMI / arXiv / Project / Code
Embodied RAG meets open-world agents.
JARVIS-1: Open-world Multi-task Agents with Memory-Augmented Multimodal Language Models
T-PAMI / arXiv / Project / Code
Embodied RAG meets open-world agents.
Haozhe Zhao*,
Xiaojian Ma*,
Liang Chen,
Shuzheng Si,
Rujie Wu
Kaikai An,
Peiyu Yu,
Minjia Zhang,
Qing Li
Baobao Chang
UltraEdit: Instruction-based Fine-Grained Image Editing at Scale
NeurIPS 2024 D&B Track / arXiv / Demo / Project / Code
Free-form and region-based image editing made easy with language.
UltraEdit: Instruction-based Fine-Grained Image Editing at Scale
NeurIPS 2024 D&B Track / arXiv / Demo / Project / Code
Free-form and region-based image editing made easy with language.
Team CraftJarvis
OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents
NeurIPS 2024 / arXiv / Project / Code
Native modeling of multimodal interaction (VLA) data.
OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents
NeurIPS 2024 / arXiv / Project / Code
Native modeling of multimodal interaction (VLA) data.
Yue Fan*,
Xiaojian Ma*,
Rujie Wu
Yuntao Du,
Jiaqi Li,
Zhi Gao,
Qing Li
VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding
ECCV 2024 / arXiv / Project
Zero-shot long-form video understanding, shrinking the gap to Gemini.
VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding
ECCV 2024 / arXiv / Project
Zero-shot long-form video understanding, shrinking the gap to Gemini.
Ziyu Zhu,
Zhuofan Zhang,
Xiaojian Ma,
Xuesong Niu,
Yixin Chen,
Baoxiong Jia,
Zhidong Deng,
Siyuan Huang,
Qing Li
Unifying 3D Vision-Language Understanding via Promptable Queries
ECCV 2024 / arXiv / Project / Code
Unifying open-vocabulary perception and reasoning in 3D world.
Unifying 3D Vision-Language Understanding via Promptable Queries
ECCV 2024 / arXiv / Project / Code
Unifying open-vocabulary perception and reasoning in 3D world.
Xiaojian Ma*,
Jiangyong Huang*,
Silong Yong*,
Xiongkun Linghu*,
Puhao Li,
Yan Wang,
Qing Li,
Song-Chun Zhu,
Baoxiong Jia,
Siyuan Huang
LEO: An Embodied Generalist Agent in 3D World
ICML 2024 / arXiv / Project / Code / Demo / Video
LEO: An Embodied Generalist Agent in 3D World
ICML 2024 / arXiv / Project / Code / Demo / Video
Ran Gong*,
Xiaojian Ma*,
Qiuyuan Huang*,
Hoi Vo,
Zane Durante
Yusuke Noda,
Zilong Zheng,
Song-Chun Zhu
Demetri Terzopoulos,
Li Fei-Fei,
Jianfeng Gao
MindAgent: Emergent Gaming Interaction
NAACL 2024 Findings / Paper / arXiv / Project / Code
Benchmark and other infra for LLM + general multi-player gaming.
MindAgent: Emergent Gaming Interaction
NAACL 2024 Findings / Paper / arXiv / Project / Code
Benchmark and other infra for LLM + general multi-player gaming.
Zhi Gao,
Yuntao Du,
Xintong Zhang,
Xiaojian Ma,
Wenjuan Han,
Song-Chun Zhu,
Qing Li,
CLOVA: A Closed-LOop Visual Assistant with Tool Usage and Update
CVPR 2024 / arXiv / Project
A self-improved language agent that sharpens its tools.
CLOVA: A Closed-LOop Visual Assistant with Tool Usage and Update
CVPR 2024 / arXiv / Project
A self-improved language agent that sharpens its tools.
Haozhe Zhao*,
Zefan Cai,
Zhuzheng Si,
Xiaojian Ma,
Kaikai An,
Liang Chen,
Zixuan Liu,
Sheng Wang,
Wenjuan Han,
Baobao Chang,
MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning
ICLR 2024 / Paper / arXiv / Demo / Code
Ranked 1st on MMBench and MME in 08/2023.
MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning
ICLR 2024 / Paper / arXiv / Demo / Code
Ranked 1st on MMBench and MME in 08/2023.
Rujie Wu*
Xiaojian Ma*,
Qing Li,
Wei Wang,
Zhenliang Zhang,
Song-Chun Zhu
Yizhou Wang
Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World
ICLR 2024 / Paper / arXiv / Project / Code
The third chapter of the Bongard trilogy, for the LM era (chapter 1, chapter 2)
Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World
ICLR 2024 / Paper / arXiv / Project / Code
The third chapter of the Bongard trilogy, for the LM era (chapter 1, chapter 2)
Team CraftJarvis
Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents
NeurIPS 2023 / Paper / arXiv / Project / Code
Best paper award, ICML-23 TEACH Workshop
Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents
NeurIPS 2023 / Paper / arXiv / Project / Code
Best paper award, ICML-23 TEACH Workshop
Ziyu Zhu,
Xiaojian Ma,
Yixin Chen,
Zhidong Deng,
Siyuan Huang,
Qing Li
3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment
ICCV 2023 / arXiv / Project / Code
A new 3D-Language foundation model
3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment
ICCV 2023 / arXiv / Project / Code
A new 3D-Language foundation model
Team CraftJarvis
Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Prediction
CVPR 2023 / arXiv / Project / Code
Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Prediction
CVPR 2023 / arXiv / Project / Code
Xiaojian Ma,
Silong Yong,
Zilong Zheng,
Qing Li,
Yitao Liang,
Song-Chun Zhu,
Siyuan Huang
SQA3D: Situated Question Answering in 3D Scenes
ICLR 2023 / Paper / arXiv / Slides / Project / Code / Benchmark
A new quest: embodied scene understanding
SQA3D: Situated Question Answering in 3D Scenes
ICLR 2023 / Paper / arXiv / Slides / Project / Code / Benchmark
A new quest: embodied scene understanding
Peyi Yu,
Sirui Xie,
Xiaojian Ma,
Baoxiong Jia,
Bo Pang,
Ruiqi Gao,
Yixin Zhu,
Song-Chun Zhu and
Ying Nian Wu
Latent Diffusion Energy-Based Model for Interpretable Text Modeling
ICML 2022 / Paper / arXiv / Code
Latent Diffusion Energy-Based Model for Interpretable Text Modeling
ICML 2022 / Paper / arXiv / Code
Huaizu Jiang*,
Xiaojian Ma*,
Weili Nie,
Zhiding Yu,
Yuke Zhu,
Song-Chun Zhu,
Anima Anandkumar
Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions
CVPR 2022 / Paper / Poster / Slides / Project / arXiv / Code / Bibtex
Oral presentation
Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions
CVPR 2022 / Paper / Poster / Slides / Project / arXiv / Code / Bibtex
Oral presentation
Xiaojian Ma,
Weili Nie,
Zhiding Yu,
Huaizu Jiang,
Chaowei Xiao,
Yuke Zhu,
Song-Chun Zhu,
Anima Anandkumar
RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning
ICLR 2022 / Paper / Poster / Slides / Project / OpenReview / arXiv / Code / Bibtex
RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning
ICLR 2022 / Paper / Poster / Slides / Project / OpenReview / arXiv / Code / Bibtex
Peyi Yu,
Sirui Xie,
Xiaojian Ma,
Yixin Zhu,
Ying Nian Wu and
Song-Chun Zhu
Unsupervised Foreground Extraction via Deep Region Competition
NeurIPS 2021 / Paper / arXiv / Code
Unsupervised Foreground Extraction via Deep Region Competition
NeurIPS 2021 / Paper / arXiv / Code
Mingxuan Jing,
Wenbing Huang,
Fuchun Sun,
Xiaojian Ma,
Tao Kong,
Chuang Gan and
Lei Li
Adversarial Option-Aware Hierarchical Imitation Learning
ICML 2021 / Paper / arXiv / Code
Spotlight presentation
Adversarial Option-Aware Hierarchical Imitation Learning
ICML 2021 / Paper / arXiv / Code
Spotlight presentation
Mark Edmonds,
Xiaojian Ma,
Siyuan Qi,
Yixin Zhu,
Hongjing Lu and
Song-Chun Zhu
Theory-based Causal Transfer: Integrating Instance-level Induction and Abstract-level Structure Learning
AAAI 2020 / Paper / Project Page / arXiv / Code
Oral presentation
Theory-based Causal Transfer: Integrating Instance-level Induction and Abstract-level Structure Learning
AAAI 2020 / Paper / Project Page / arXiv / Code
Oral presentation
Xiaojian Ma*,
Mingxuan Jing*,
Wenbing Huang,
Fuchun Sun,
Bin Fang and
Huaping Liu
Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance
AAAI 2020 / Paper / Project Page / arXiv / Code
also in Structure & Priors in Reinforcement Learning Workshop @ ICLR 2019
Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance
AAAI 2020 / Paper / Project Page / arXiv / Code
also in Structure & Priors in Reinforcement Learning Workshop @ ICLR 2019
Xiaojian Ma*,
Chao Yang*,
Wenbing Huang*,
Fuchun Sun,
Huaping Liu,
Junzhou Huang and
Chuang Gan,
Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement
NeurIPS 2019 / Paper / Project Page / arXiv / Code
Spotlight presentation
Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement
NeurIPS 2019 / Paper / Project Page / arXiv / Code
Spotlight presentation
Hongzhuo Liang,
Shuang Li,
Xiaojian Ma,
Norman Hendrich,
Timo Gerkmann,
Fuchun Sun and
Jianwei Zhang
Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring
IROS 2019 / Paper / Project Page / arXiv / Code / Video
Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring
IROS 2019 / Paper / Project Page / arXiv / Code / Video
Xiaojian Ma*,
Hongzhuo Liang*,
Shuang Li,
Michael Görner,
Song Tang,
Bin Fang,
Fuchun Sun and
Jianwei Zhang
PointNetGPD: Detecting Grasp Configurations from Point Sets
ICRA 2019 / Paper / Project Page / arXiv / Code / Video
PointNetGPD: Detecting Grasp Configurations from Point Sets
ICRA 2019 / Paper / Project Page / arXiv / Code / Video
Xiaojian Ma*,
Shuang Li*,
Hongzhuo Liang,
Michael Görner,
Philipp Ruppel,
Bin Fang,
Fuchun Sun and
Jianwei Zhang
Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network
ICRA 2019 / Paper / Project Page / arXiv / Code / Video
Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network
ICRA 2019 / Paper / Project Page / arXiv / Code / Video
Xiaojian Ma*,
Mingxuan Jing*,
Wenbing Huang,
Fuchun Sun and
Huaping Liu
Task Transfer by Preference-Based Cost Learning
AAAI 2019 / Paper / Project Page / arXiv / Code
Spotlight presentation
Task Transfer by Preference-Based Cost Learning
AAAI 2019 / Paper / Project Page / arXiv / Code
Spotlight presentation
Experience
-
2022.9 - 2023.1, Research Scientist Intern, DeepMind
Mentors: Kory Mathewson, Peter Humphreys, Owen He, Adam Santoro, Timothy Lillicrap and Doina Precup. -
2021.6 - 2021.12, Research Intern, NVIDIA Research
Mentors: Weili Nie, Huaizu Jiang, Chaowei Xiao, Zhiding Yu, Yuke Zhu and Anima Anandkumar. -
2020.6 - 2021.5, Research Intern, Google Brain Robotics
Mentors: Pannag Sanketi and Laura Graesser. -
2019.6 - 2019.9, Research Intern, ByteDance AI Lab
Mentors: Tao Kong and Lei Li. -
2017.7 - 2017.9, Research Intern, CVRP Lab, School of Computer Science, National University of
Singapore
Mentor: Gim Hee Lee.
© Xiaojian Ma 2024