I am joining the Computer Science Department at Johns Hopkins University as an Assistant Professor in Fall 2026. Until that, I am spending my gap year at AI2 Fall 2025 - Fall 2026.

I will be recruiting Ph.D. students for Fall 2026. Please see this page for more details. I plan to attend ICCV 2025 (in Hawaii) and NeurIPS 2025 (in San Diego). Please feel free to reach out if you would like to chat in person!

My research focuses on multimodal AI, integrating diverse data types (e.g., images, videos, text, audio, and motion) to develop models that are interpretable, controllable, and scalable. My recent interests include learning action knowledge (e.g., robot actions, human actions, physical laws) from unlabeled videos, learning to reason in non-textual chain-of-thoughts at scale, and developing AI-based softwares that enhance human productivity in various applications (e.g., film, design, music, choreography, medicine).

Below is a summary of my research during my PhD years.

(1) Scalable Multimodal Frameworks – Modern AI models must meet the growing demand for thousands of capabilities. My research has addressed this challenge by introducing: (a) Unified generative frameworks that flexibly accommodate diverse modalities and tasks, using a single architecture and a generative objective – VL-T5 (ICML 2021) / X-LXMERT (EMNLP 2020) / TVLT (NeurIPS 2022 Oral) and (b) Efficient finetuning frameworks that significantly reduce parameter and memory requirements for creating task-specific models – VL-Adapter (CVPR 2022) / LST (NeurIPS 2022) / Ctrl-Adapter (ICLR 2025 Oral)

(2) Faithful Multimodal Reasoning – Scaling alone is not enough. Large models that rely on black-box reasoning and encode all knowledge within their parameters often struggle with basic tasks and produce hallucinations. My research makes their reasoning process more accurate and interpretable by introducing: (a) Planning-based frameworks that decompose complex visual generation problems into faithful, human-interpretable step-by-step reasoning processes – VPGen (NeurIPS 2023) / VideoDirectorGPT (COLM 2024) / DiagrammerGPT (COLM 2024) / Video-MSG(2024) and (b) Retrieval-augmented generation (RAG) frameworks that enhance accuracy and factuality by retrieving relevant information before generating outputs – M3DocRAG (2024) / HiREST (CVPR 2023)

(3) Evaluation and Refinement of Multimodal Generation – With recent advancements in multimodal generation models, conventional evaluation metrics have been often saturated and no longer provide meaningful insights into future research direction. To this end, my research introduces: (a) Fine-grained evaluation frameworks that comprehensively measure model skills in multiple dimensions to uncover detailed strengths and weaknesses – DALL-Eval (ICCV 2023) / VPEval (NeurIPS 2023) / DSG (ICLR 2024) / LayoutBench (CVPRW 2024 Oral) / FineCapEval (Findings of NAACL 2022) / M3DocVQA (2024) / CAPTURe (ICCV 2025) and (b) Automatic model refinement frameworks that use these evaluations to detect models’ weaknesses and refine their reasoning process – EnvGen (COLM 2024) / DataEnvGym (ICLR 2025 Spotlight) / SELMA (NeurIPS 2024) / VideoRepair (2024)

I’m also sharing some of my past application materials below. I know these are far from perfect, but I hope this helps you in your applications!

• Academic Job Market (written in Dec 2024): Research Statement, Teaching Statement, DEI Statement
• Bloomberg Data Science Fellowship (written in Apr 2023): Research Statement
• PhD Admission (written in Dec 2019): Statement of Purpose.
  • This collection of SOPs for CS PhD programs looks useful as well!

News

• Sep 2025 - 1 paper accepted at NeurIPS 2025:

  • Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents

• Sep 2025 - I’m starting my gap year at AI2 PRIOR team as an Young Investigator!

• Aug 2025 - 1 paper accepted at Findings in EMNLP 2025:

  • Video-Skill-CoT: Skill-based Chain-of-Thoughts for Domain-Adaptive Video Reasoning

• Aug 2025 - New preprints:

  • Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents
  • RotBench: Evaluating Multimodal Large Language Models on Identifying Image Rotation

• Jul 2025 - New preprints:

  • A Survey on Long-Video Storytelling Generation: Architectures, Consistency, and Cinematic Quality

• Jun 2025 - 1 paper accepted at ICCV 2025:

  • CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via Occluded Object Counting

• Jun 2025 - New preprints:

  • CLaMR: Contextualized Late-Interaction for Multimodal Content Retrieval
  • Video-Skill-CoT: Skill-based Chain-of-Thoughts for Domain-Adaptive Video Reasoning

• May 2025 - New preprints:

  • EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidance

• Apr 2025 - New preprints:

  • CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via Occluded Object Counting
  • Executable Functional Abstractions: Inferring Generative Programs for Advanced Math Problems
  • Video-MSG: Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initialization

• Feb 2025 - 2 papers have been accepted to ICLR 2025:

  • Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model
    • as Oral presentation (top 1.8% of 10,000+ submissions)
  • DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback
    • as Spotlight presentation (top 5.1% of 10,000+ submissions)