Research Interests

MLLM | LLM | MoE | Model Architecture | Retrieval & Embedding | Domain Adaption | Multilingual | Text Generation

Selected Publications

• Multimodal Latent Language Modeling with Next-Token Diffusion
  Yutao Sun, Hangbo Bao, Wenhui Wang, Zhiliang Peng, Li Dong, Shaohan Huang, Jianyong Wang, Furu Wei. arxiv. MLLM
• Textual Aesthetics in Large Language Models
  Lingjie Jiang#, Shaohan Huang, Xun Wu, Furu Wei. arxiv. LLM
• DeepNet: Scaling Transformers to 1,000 Layers
  Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Furu Wei. PAMI 2024. Model Architecture
• Text Diffusion with Reinforced Conditioning
  Yuxuan Liu#, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang. AAAI 2024.Text Generation
• Se2: Sequential Example Selection for In-Context Learning
  Haoyu Liu#, Jianfeng Liu, Shaohan Huang, Yuefeng Zhan, Hao Sun, Weiwei Deng, Furu Wei, Qi Zhang. ACL 2024. Retrieval & Embedding LLM
• HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition
  Yuxuan Liu#, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang. ACL 2024. LLM
• ResLoRA: Identity Residual Mapping in Low-Rank Adaption
  Shuhua Shi#, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang. ACL 2024. LLM
• Calibrating LLM-Based Evaluator
  Yuxuan Liu#, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang. COLING 2024. LLM
• Instruction Pre-Training: Language Models are Supervised Multitask Learners
  Daixuan Cheng#, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, Furu Wei. EMNLP 2024. LLM
• Scaling Sentence Embeddings with Large Language Models
  Ting Jiang#, Shaohan Huang, Zhongzhi Luan, Deqing Wang, Fuzhen Zhuang. EMNLP 2024. LLM Retrieval & Embedding
• Adapting Large Language Models via Reading Comprehension
  Daixuan Cheng#, Shaohan Huang, Furu Wei. ICLR 2024.Domain Adaption
• Kosmos-G: Generating Images in Context with Multimodal Large Language Models
  Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei. ICLR 2024. MLLM
• Grounding Multimodal Large Language Models to the World
  Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, Furu Wei. ICLR 2024. MLLM
• Mixture of LoRA Experts
  Xun Wu#, Shaohan Huang, Furu Wei. ICLR 2024. LLM
• MoEC: Mixture of Expert Clusters
  Yuan Xie#, Shaohan Huang, Tianyu Chen, Furu Wei. AAAI 2023. MoE
• Pre-training Language Model as a Multi-perspective Course Learner
  Beiduo Chen#, Shaohan Huang, Zihan Zhang, Wu Guo, Zhenhua Ling, Haizhen Huang, Furu Wei, Weiwei Deng, Qi Zhang. ACL 2023. LLM
• Dual-Alignment Pre-training for Cross-lingual Sentence Embedding
  Ziheng Li#, Shaohan Huang, Zihan Zhang, Zhi{-}Hong Deng, Qiang Lou, Haizhen Huang, Jian Jiao, Furu Wei, Weiwei Deng, Qi Zhang. ACL 2023. Retrieval & Embedding
• Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
  Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song. ACL 2023. Multilingual
• Democratizing Reasoning Ability: Tailored Learning from Large Language Model
  Zhaoyang Wang#, Shaohan Huang, Yuxuan Liu, Jiahai Wang, Minghui Song, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang. EMNLP 2023. LLM
• UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation
  Daixuan Cheng#, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Weiwei Deng, Qi Zhang. EMNLP 2023. Retrieval & Embedding
• Magneto: A Foundation Transformer
  Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, Barun Patra, Zhun Liu, Vishrav Chaudhary, Xia Song, Furu Wei. ICML 2023. Model Architecture
• Language Is Not All You Need: Aligning Perception with Language Models
  Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Nils Johan Bertil Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei. NIPS 2023. MLLM
• THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
  Tianyu Chen#, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, Furu Wei. ACL 2022. Encryption
• XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
  Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Bo Zheng, Saksham Singhal, Payal Bajaj, Xia Song, Xianling Mao, Heyan Huang, Furu Wei. ACL 2022. Multilingual MoE
• CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translation
  Jian Yang, Shaohan Huang, Shuming Ma, Yuwei Yin, Li Dong, Dongdong Zhang, Hongcheng Guo, Zhoujun Li, Furu Wei. EMNLP 2022. Multilingual
• Snapshot-Guided Domain Adaptation for ELECTRA
  Daixuan Cheng#, Shaohan Huang, Jianfeng Liu, Yuefeng Zhan, Hao Sun, Furu Wei, Denvy Deng, Qi Zhang. EMNLP 2022. Domain Adaption
• PromptBERT: Improving BERT Sentence Embeddings with Prompts
  Ting Jiang#, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, Qi Zhang. EMNLP 2022. Retrieval & Embedding
• On the Representation Collapse of Sparse Mixture of Experts
  Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xianling Mao, Heyan Huang, Furu Wei. NIPS 2022.MoE
• Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
  Yunzhi Yao#, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei. ACL 2021. Domain Adaption
• Consistency Regularization for Cross-Lingual Fine-Tuning
  Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei. ACL 2021. Multilingual
• Allocating Large Vocabulary Capacity for Cross-Lingual Language Model Pre-Training
  Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei. EMNLP 2021. Multilingual
• Unsupervised Fine-tuning for Text Clustering
  Shaohan Huang, Furu Wei, Lei Cui, Xingxing Zhang, Ming Zhou. COLING 2020. Retrieval & Embedding
• Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph
  Haozhe Ji#, Pei Ke, Shaohan Huang, Furu Wei, Xiaoyan Zhu, Minlie Huang. EMNLP 2020.Text Generation
• Generating Commonsense Explanation by Extracting Bridge Concepts from Reasoning Paths
  Haozhe Ji#, Pei Ke, Shaohan Huang, Furu Wei, Minlie Huang. IJCNLP 2020.Text Generation
• Dictionary-Guided Editing Networks for Paraphrase Generation
  Shaohan Huang, Yu Wu, Furu Wei, Zhongzhi Luan. AAAI 2019.Text Generation
• Learning to Generate Product Reviews from Attributes
  Li Dong, Shaohan Huang, Furu Wei, Mirella Lapata, Ming Zhou, Ke Xu. EACL 2017.Text Generation

Students

• Zhaoyang Wang (Master, Sun Yat-sen University)
• Lingwei Wei (PhD, Chinese Academy of Sciences)
• Chang Ma (undergraduate, Peking University)
• Ting Jiang (PhD, Beihang University)
• Yunzhi Yao (PhD, Zhejiang University)
• Haozhe Ji (PhD, Tsinghua University)
• Tianyu Chen (PhD, Beihang University)

Readings

On Being a Scientist: A Guide to Responsible Conduct in Research National Academies of Sciences, Engineering, and Medicine

How to Write a Lot: A Practical Guide to Productive Academic Writing Paul J. Silvia

How to Write a Great Research Paper Simon Peyton Jones

How to Give a Good Research Talk Simon Peyton Jones

An Awesome Blog with Many Suggestions for Graduate Students Matt Might

A Research to Engineering Workflow Dustin Tran

Heuristics for Scientific Writing (a Machine Learning Perspective) Zachary C. Lipton