Robotic Perception, Interaction, and Learning Lab (RoboPIL)

• At RoboPIL, we work at the intersection of robotics, computer vision, and machine learning. Specifically, we focus on Robot Learning and aim to significantly expand robots' perception and physical interaction capabilities, particularly through the following three directions.
  • Structured World Models: Learning physics-inspired predictive models from and for robotic manipulation of deformable objects.
  • Embodied Intelligence: Developing and integrating robotic foundation models for generalizable and long-horizon embodied interactions.
  • Multi-Modal Perception: Harnessing vision, touch, audio, and language for fine-grained and effective manipulation.
• Prospective Students: I'm looking for technically strong and self-motivated students passionate about advancing the frontiers of robot learning.
  • PhD Applicants: Please submit your application to the CS PhD program, and mention me as one of your Faculty of Interest.
  • Postdoc and Visiting Students: Please reach out to me directly via email, attaching your CV.


• PhD Students
  
  
  
  Binghao Huang

  
  Hanxiao Jiang
  (w/ Shenlong Wang)

  
  Shivansh Patel
  (w/ Svetlana Lazebnik)

  
  Rhythm Syed
  (w/ Tony Dear)

  
  Yixuan Wang

  
  Zach Witzel

  
  Kaifeng Zhang


• Master's and Undergraduate Students
  • Yuan Fang (Master)
  • Shuo (Chris) Sha (CS Undergrad)

  • Naian Tao (ME Master)
  • Xinyue (Yolanda) Zhu (CS Undergrad)

Recent Talks

• [2026/03] Invited Talks at Princeton Robotics Seminar and Johns Hopkins University LCSR Seminar
• [2026/02] Invited Talks at Harvard Vision AI Seminar, Yale University, and Mila Robot Learning Seminar
• [2025/12] Invited Talk at OpenAI
• [2025/11] Invited Talks at Brown University and Lehigh University
• [2025/11] Guest Lecture at Physical Intelligence at UPenn
• [2025/10] Invited Talk at IROS 2025 Workshop on Advanced Haptic Sensors and Devices for Healthcare Robots
• [2025/10] Invited Talk at ICCV 2025 Workshop on Learning to See: Advancing Spatial Understanding for Embodied Intelligence
• [2025/10] Invited Talk at ICCV 2025 Tutorial on Foundation Models Meet Embodied Agents
• [2025/10] Invited Talk at Georgia Tech IRIM Robotics Seminar
• [2025/09] Invited Talk at CoRL 2025 Workshop on Robotics World Modeling
• [2025/09] Invited Talk at CoRL 2025 Workshop on Beyond Rigid Worlds: Representing and Interacting with Non-Rigid Objects
• [2025/09] Invited Talk at Toyota Research Institute (TRI)'s ML Speaker Series
• [2025/07] Invited Talk at SFU @ ICML 2025 at Simon Fraser University
• [2025/07] Invited Talk at ACC 2025 Workshop on LLMs in Control Design and Decision Making
• [2025/06] Invited Talk at RSS 2025 Workshop on Robot Planning in the Era of Foundation Models
• [2025/06] Invited Talk at RSS 2025 Workshop on Structured World Models for Robotic Manipulation
• [2025/06] Invited Talk at RSS 2025 Workshop on MultiSensory Robotics with MultiModal Capabilities
• [2025/06] Invited Talk at RSS 2025 Workshop on Large Foundation Model for Interactive Robot Learning
[2025/06] Invited Talk at CVPR 2025 Workshop on 3D Vision Language Models (VLMs) for Robotics Manipulation
[2025/06] Invited Talk at CVPR 2025 Workshop on Generalization in Robotics Manipulation Workshop and Challenges
[2025/06] Invited Talk at CVPR 2025 Workshop on Computer Vision in the Wild
[2025/06] Keynote at the New England Manipulation Symposium (NEMS)
[2025/05] Guest Lecture at CS231n: Deep Learning for Computer Vision at Stanford [Recording]
[2025/05] Invited Talk at ICRA 2025 Workshop on Reflections on Representations and Manipulating Deformable Objects [Recording]
[2025/05] Invited Talk at ICRA 2025 Workshop on Foundation Models and Neuro-Symbolic AI for Robotics [Recording]
[2025/05] Invited Talk at ICRA 2025 Workshop on Structured Learning for Efficient, Reliable and Transparent Robots
[2025/05] Invited Talk at Waymo and Microsoft Cortex AI Series
[2025/05] Invited Talk at NYC Robotics Mayor’s Cup
[2025/05] Invited Talk at NAACL 2025 Tutorial on Foundation Models Meet Embodied Agents
[2025/04] Invited Talk at National University of Singapore (NUS) [Recording]
[2025/04] Invited Talk at UT Austin Robot Learning and Manipualtion Seminar
[2025/04] Invited Talk at Princeton University
[2025/04] Invited Talk at MSR-NYC Agent Representation Learning Workshop
[2025/04] Guest Lecture at 16-831: Introduction to Robot Learning at CMU [Recording]
[2025/03] Invited Talk at Google DeepMind
[2025/03] Invited Talk at UMich Robotics Seminar [Recording]
[2025/03] Invited Talk at Columbia AI Summit [Recording]
[2025/02] Invited Talk at Samsung Research
[2025/02] Talks at AAAI 2025 Tutorial on Foundation Models Meet Embodied Agents [Recording] and New Faculty Highlights
[2025/02] Guest Lecture at COMS 6113: Agentic Systems Made Real at Columbia University
[2025/02] Keynote at NYC Computer Vision Day
[2025/01] Invited Talks at Peking University and Tsinghua University
[2024/11] Invited Talk at UPenn GRASP Seminar
[2024/11] Invited Talk at CoRL 2024 Workshop on Differentiable Optimization Everywhere
[2024/11] Invited Talk at Next-Gen Robot Learning Symposium at TU Darmstadt
[2024/09] Invited Talk at Allerton Conference on Communication, Control, and Computing
[2024/08] Invited Talk at TTIC Summer Workshop on Multimodal Artificial Intelligence
[2024/07] Invited Talk at RSS 2024 Workshop on Structural Priors as Inductive Biases for Learning Robot Dynamics
[2024/07] Invited Talk at RSS 2024 Workshop on Koopman Operators in Robotics
[2024/06] Invited Talk at Boston Dynamics AI Institute
[2024/05] Invited Talk at ICRA 2024 Workshop on A Future Roadmap for Sensorimotor Skill Learning for Robot Manipulation
[2024/05] Invited Talk at ICRA 2024 Workshop on Bimanual Manipulation: On Kitchen Challenges
[2024/03] Invited Talk at CMU Robotics Institute, VASC Seminar [Recording]
[2024/03] Guest Lecture at 16-831: Introduction to Robot Learning at CMU [Recording]
[2024/03] Invited Talk at AICE Spring Research Symposium
[2024/01] Invited Talk at Mila Robot Learning Seminar [Recording]
[2023/12] Invited Talk at Amazon Science
[2023/11] Invited Talk at Tsinghua University, Institute for Interdisciplinary Information Sciences (IIIS)
[2023/10] Invited Talk at IROS 2023 Workshop on Causality for Robotics
[2023/09] Invited Talk at Robotics Seminar @ Illinois
[2023/08] Invited Talk at Stanford Intelligent Systems Laboratory
[2023/07] Invited Talk at Peking University
[2023/06] Invited Talk at CVPR 2023 Workshop on Precognition: Seeing through the Future [Recording]
[2023/05] Invited Talk at ICRA 2023 Workshop on Representing and Manipulating Deformable Objects
[2022/12] Guest Lecture at NYU, Invited Talk at Shanghai AI Laboratory
[2022/10] Invited Talk at Stanford and UC Berkeley
[2022/07] Invited Talk at RSS 2022 Workshop on Implicit Representations for Robotic Manipulation [Recording] and ISEE AI
[2022/06] PhD Thesis Defense at MIT CSAIL [Recording]
[2022/04] Invited Talk at Google Research
[2022/01] Invited Talk at TU Berlin
[2021/12] Invited Talk at Meta AI Research
[2021/11] Invited Talk at Toyota Research Institute
[2021/08] Invited Talk at Wayve.ai and Tsinghua University
[2021/06] Invited Talk at CVPR 2021 Tutorial on Learning Representations via Graph-structured Networks [Recording]
[2021/05] Invited Talk at ICLR 2021 Workshop on Deep Learning for Simulation [Recording]
[2021/04] Invited Talk at Brown University
[2021/03] Invited Talk at Extrality.ai
[2020/10] Invited Talk at Stanford and UMass Amherst
[2020/09] Invited Talk at the University of Toronto [Recording]
• [Show more]

Publications (show selected / show by date / show by topic)
Research Topics: Robotic Manipulation / Foundation Models for Embodied Agents / Physical Scene Understanding / Multi-Modal Perception

Selected Honors

• AAAI New Faculty Highlights, 2025 [Link]
• Amazon Research Award, Fall 2024 [Link]
• Teachers Ranked as Excellent at UIUC, Spring 2024 [Link]
• Best Conference Paper Award, International Conference on Robotics and Automation (ICRA) 2024 [Link]
• Sony Faculty Innovation Award, 2023 [Link]
• Best Systems Paper Award, Conference on Robot Learning (CoRL) 2023 [Link]
• Finalist, Best Paper/Best Student Paper Awards, Conference on Robot Learning (CoRL) 2023 [Link]
• First Place Recipient, Ernst A. Guillemin Master's Thesis Award in Artificial Intelligence and Decision Making at MIT, 2021 [Link]
• Adobe Research Fellowship, 2020 [Link]
• Outstanding Undergraduate Thesis Award, EECS Dept., Peking Univ. (10 in 400)

Professional Service

• Workshop Organizer
• Learning to Simulate Robot Worlds at CoRL 2025
• Structured World Models for Robotic Manipulation at RSS 2025
• Foundation Models Meet Embodied Agents at CVPR 2025
• Foundation Models and Neuro-Symbolic AI for Robotics at ICRA 2025
• How Far Are We From AGI? at ICLR 2024
• Implicit Representations for Robotic Manipulation at RSS 2022
• Multi-Agent Interaction and Relational Reasoning at ICCV 2021
• Deformable Object Simulation in Robotics at RSS 2021
• Vision Meets Cognition at CVPR 2019
• Tutorial Organizer
• Foundation Models Meet Embodied Agents at AAAI 2025, NAACL 2025, ICCV 2025
• Learning Representations via Graph-structured Networks at CVPR 2021
• Multi-Modality Learning from Videos and Beyond at ICCV 2021
• Area Chair: CoRL, CVPR, ICLR, RSS
• Associate Editor: IEEE RA-L, IROS
• Senior Program Committee Member: AAAI
• Conference Reviewer: ICLR, ICML, NeurIPS, ICCV, CVPR, ECCV, ICRA, IROS, RSS, CoRL, Humanoids, SIGGRAPH Asia
• Program Committee Member: AAAI, UAI
• Journal Reviewer: Science, Nature Electronics, Science Robotics, Nature Communications, TPAMI, IJCV, IJRR, IEEE T-RO, IEEE RA-L, IEEE-TMI, IEEE T-ITS, IEEE Access, IEEE-ToH