Lu Sheng · 盛律

School of Software, Beihang University

prof_pic_v2.png

I am a Professor at the School of Software, Beihang University (BUAA), China.

Before I joined BUAA in 2019, I was a postdoctoral researcher (2017-2019) in MMLab@CUHK, working with Prof. Xiaogang Wang. I received my Ph.D. from Department of Electronic Engineering at the Chinese University of Hong Kong (CUHK), advised by Prof. King Ngi Ngan (IEEE Life Fellow).

My research interests include 3D computer vision and embodied AI. My current research lies in developing generalizable models for understanding, interacting with and synthesizing 3D visual world.

To Prospective Students: I am actively looking for highly-motivated students targeted to Master or Ph.D. degree, as well as undergraduate-level research assistants. Please drop me an email with your resume if you are interested in my research.

news

Jul 06, 2025 Multi-Agent Amodal Completion has been accepted at ACM MM 2025!
Jun 26, 2025 MV-Adapter has been accepted at ICCV 2025! Please check out the code, models and huggingface demos!
Jun 16, 2025 RH20T-P and MineDreamer have been accepted at IROS 2025! Please check out the code and demos!
Mar 28, 2025 MIDI, Code-as-Monitor, Ouroboros3D, and T2ISafety has been accepted at CVPR 2025! Please check out the demos in their webpages!
Sep 14, 2024 I will serve as Area Chair of CVPR 2025.
Jun 24, 2024 SketchSampler++ (the extension of the previous ECCV 2022 paper SketchSampler) is accepted at IEEE T-PAMI! Congratulations to all co-authors！
Jun 15, 2024 Fast-BEV is finally accepted at IEEE T-PAMI! Congratulations to all co-authors！
Feb 27, 2024 MP5 and EpiDiff has been accepted at CVPR 2024! Please check out the demos in their webpages!
Jan 17, 2024 I will serve as Area Chair of ECCV 2024.
Jan 16, 2024 Our work Octavius has been accepted at ICLR 2024! It is one of the first mixture-of-experts (MoE) papers about MLLMs. All data and code are now open-sourced, stay tuned for updated!

selected publications

  1. ICCV
    iccv25_mvadapter.png
    MV-Adapter: Multi-view Consistent Image Generation Made Easy
    Zehuan Huang, Yuan-Chen Guo , Haoran Wang, Ran Yi, Yangguang Li, Lizhuang Ma, Yan-Pei Cao*, and Lu Sheng*
    In IEEE/CVF International Conference on Computer Vision (ICCV) , Oct 2025
    arXiv Code Website
  2. CVPR
    cvpr25_cam.png
    Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection
    Enshen Zhou#, Qi Su#, Cheng Chi#* , Zhizheng Zhang , Zhongyuan Wang, Tiejun Huang, Lu Sheng* , and He Wang*
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , Jun 2025
    PDF Video Website
  3. CVPR
    cvpr25_midi.png
    MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation
    Zehuan Huang, Yuan-Chen Guo, Xingqiao An, Yunhan Yang, Yangguang Li, Zi-Xin Zou, Ding Liang, Xihui Liu, Yan-Pei Cao*, and Lu Sheng*
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , Jun 2025
    Code Website
  4. CVPR
    neurips24_o3d.png
    Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion
    Hao Wen#, Zehuan Huang# , Yaohui Wang, Xinyuan Chen, and Lu Sheng*
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , Jun 2025
    arXiv PDF Code Website
  5. IEEE T-IP
    tip25_p2w.png
    Parts2Whole: Generalizable Multi-Part Portrait Customization
    Zehuan Huang#, Hongxin Fan# , Lipeng Wang, Haohua Chen , Li Yin, and Lu Sheng*
    IEEE Transactions on Image Processing, Jun 2025
    arXiv HTML Code Website
  6. IROS
    eccv24_minedreamer.jpg
    MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control
    Enshen Zhou#, Yiran Qin#, Zhenfei Yin, Yuzhou Huang , Ruimao Zhang*, Lu Sheng*, Yu Qiao, and Jing Shao
    In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , Jun 2025
    arXiv Code Website
  7. IROS
    eccv24_rh20tp.png
    RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agents
    Zeren Chen, Zhelun Shi , Xiaoya Lu, Lehan He, Sucheng Qian, Haoshu Fang, Zhenfei Yin, Wanli Ouyang, Jing Shao, Yu Qiao, and 2 more authors
    In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , Jun 2025
    arXiv Website
  8. IEEE T-PAMI
    pami24_fast-bev++.png
    Fast-BEV: A Fast and Strong Bird’s-Eye View Perception Baseline
    Yangguang Li, Bin Huang, Zeren Chen, Yufeng Cui, Feng Liang, Mingzhu Shen , Fenggang Liu, Enze Xie, Lu Sheng*, Wanli Ouyang, and 1 more author
    IEEE Trans. Pattern Anal. Mach. Intell., Jun 2024
    arXiv Code
  9. IEEE T-PAMI
    pami24_sketchsampler++.png
    3D Reconstruction from a Single Sketch via View-dependent Depth Sampling
    Chenjian Gao# , Xilin Wang#, Qian Yu*, Lu Sheng, Jing Zhang, Xiaoguang Han, Yi-Zhe Song, and Dong Xu
    IEEE Trans. Pattern Anal. Mach. Intell., Jun 2024
    arXiv Code
  10. CVPR
    cvpr24_epidiff.png
    EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion
    Zehuan Huang#, Hao Wen#, Junting Dong# , Yaohui Wang, Yangguang Li, Xinyuan Chen, Yan-Pei Cao, Ding Liang, Yu Qiao, Bo Dai*, and 1 more author
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition , Jun 2024
    arXiv PDF Code Website
  11. CVPR
    cvpr24_mp5.png
    MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception
    Yiran Qin#, Enshen Zhou# , Qichang Liu#, Zhenfei Yin, Lu Sheng* , Ruimao Zhang*, Yu Qiao, and Jing Shao
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition , Jun 2024
    arXiv PDF Code Website
  12. ICLR
    iclr24_octavius.png
    Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE
    Zeren Chen# , Ziqin Wang# , Zhen Wang , Huayang Liu, Zhenfei Yin , Si Liu, Lu Sheng*, Wanli Ouyang, Yu Qiao, and Jing Shao*
    In International Conference on Learning Representations , Jun 2024
    arXiv PDF Code Website
  13. NeurIPS
    neurips23_lamm.png
    LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark
    Zhenfei Yin# , Jiong Wang#, Jianjian Cao#, Zhelun Shi# , Dingning Liu, Mukai Li, Xiaoshui Huang , Zhiyong Wang, Lu Sheng, Lei Bai*, and 2 more authors
    In Advances in Neural Information Processing Systems , Jun 2023
    arXiv HTML Video Website
  14. CVPR
    cvpr23_siamese_detr.png
    Siamese DETR
    Zeren Chen#, Gengshi Huang#, Wei Li, Jianing Teng , Kun Wang, Jing Shao, Chen Change Loy, and Lu Sheng*
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition , Jun 2023
    PDF Code
  15. CVPR
    cvpr23_vlsat.png
    VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloud
    Ziqin Wang, Bowen Cheng, Lichen Zhao, Dong Xu, Yang Tang*, and Lu Sheng*
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition (Highlight Poster) , Jun 2023
    PDF Code
  16. CVPR
    cvpr22_3djcg.png
    3DJCG: A Unified Framework for Joint Dense Captioning and Visual Grounding on 3D Point Clouds
    Daigang Cai, Lichen Zhao, Jing Zhang*, Lu Sheng, and Dong Xu
    In IEEE/CVF Conference on Computer Vision and Pattern Recognition (Oral Presentation) , Jun 2022
    PDF Code
  17. AAAI
    aaai22_danceformer.png
    DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer
    Buyu Li, Yongchi Zhao, Zhelun Shi, and Lu Sheng*
    In Thirty-Sixth AAAI Conference on Artificial Intelligence , Jun 2022
    HTML Code
  18. CVPR
    cvpr21_forgerynet.png
    ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis
    Yinan He#, Bei Gan#, Siyu Chen# , Yichun Zhou# , Guojun Yin, Luchuan Song, Lu Sheng, Jing Shao* , and Ziwei Liu
    In IEEE Conference on Computer Vision and Pattern Recognition (Oral Presentation) , Jun 2021
    PDF Code Website
  19. CVPR
    cvpr21_brnet.png
    Back-Tracing Representative Points for Voting-Based 3D Object Detection in Point Clouds
    Bowen Cheng, Lu Sheng*, Shaoshuai Shi, Ming Yang, and Dong Xu
    In IEEE Conference on Computer Vision and Pattern Recognition , Jun 2021
    PDF Code
  20. ICCV
    iccv21_3dvg.png
    3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds
    Lichen Zhao, Daigang Cai, Lu Sheng*, and Dong Xu
    In IEEE/CVF International Conference on Computer Vision (1st place at 3D Object Localization Challenge at the CVPR 2021, 1st Workshop on Language for 3D Scenes) , Jun 2021
    PDF Code
  21. ACM MM
    mm21_votehmr.png
    VoteHMR: Occlusion-Aware Voting Network for Robust 3D Human Mesh Recovery from Partial Point Clouds
    Guanze Liu, Yu Rong, and Lu Sheng*
    In Proceedings of the 29th ACM International Conference on Multimedia (Oral Presentation) , Jun 2021
    arXiv HTML Code
  22. IJCV
    ijcv20_highquality.png
    High-Quality Video Generation from Static Structural Annotations
    Lu Sheng#*, Junting Pan#, Jiaming Guo, Jing Shao, and Chen Change Loy
    Int. J. Comput. Vis., Jun 2020
    HTML Code
  23. AAAI
    aaai20_msn.png
    Morphing and Sampling Network for Dense Point Cloud Completion
    Minghua Liu, Lu Sheng, Sheng Yang, Jing Shao, and Shi-Min Hu
    In The Thirty-Fourth AAAI Conference on Artificial Intelligence , Jun 2020
    PDF Code
  24. ECCV
    eccv20_f3net.png
    Thinking in Frequency: Face Forgery Detection by Mining Frequency-Aware Clues
    Yuyang Qian , Guojun Yin, Lu Sheng*, Zixuan Chen, and Jing Shao
    In European Conference on Computer Vision , Jun 2020
    arXiv PDF
  25. IEEE T-PAMI
    tpami19_visibility.png
    Visibility Constrained Generative Model for Depth-Based 3D Facial Pose Tracking
    Lu Sheng, Jianfei Cai, Tat-Jen Cham, Vladimir Pavlovic, and King Ngi Ngan
    IEEE Trans. Pattern Anal. Mach. Intell., Jun 2019
    arXiv HTML
  26. CVPR
    cvpr19_sdgan.png
    Semantics Disentangling for Text-To-Image Generation
    Guojun Yin , Bin Liu, Lu Sheng* , Nenghai Yu, Xiaogang Wang, and Jing Shao
    In IEEE Conference on Computer Vision and Pattern Recognition (Oral Presentation) , Jun 2019
    PDF