| CARVIEW |
Meta Learning and Its Applications to Natural Language Processing
Workshop at ACL 2021
Description
Deep learning based natural language processing (NLP) has become the mainstream of research in recent years and significantly outperforms conventional methods. However, deep learning models are notorious for being data and computation hungry. These downsides limit such models' application from deployment to different domains, languages, countries, or styles, since collecting in-genre data and model training from scratch are costly. The long-tail nature of human language makes challenges even more significant.
Meta-learning, or ‘Learning to Learn’, aims to learn better learning algorithms, including better parameter initialization, optimization strategy, network architecture, distance metrics, and beyond. Meta-learning has been shown to allow faster fine-tuning, converge to better performance, and achieve outstanding results for few-shot learning in many applications. Meta-learning is one of the most important new techniques in machine learning in recent years, but the method is mainly investigated with applications in computer vision. It is believed that meta-learning has excellent potential to be applied in NLP, and some works have been proposed with notable achievements in several relevant problems, e.g., relation extraction, machine translation, and dialogue generation and state tracking. However, it does not catch the same level of attention as in the image processing community.
This workshop (Meta Learning and Its Applications to Natural Language Processing Workshop, or MetaNLP) will bring concentrated discussions on meta-learning for the field of NLP via several invited talks, oral and poster sessions with high-quality papers, and a panel of leading researchers from industry and academia. Alongside research work on new meta-learning methods, data, applications, and results, this workshop will call for novel work on understanding, analyzing, and comparing different meta-learning approaches for NLP. The workshop aims to:
- Review existing and inspire new meta-learning methods and results
- Motivate the application of meta-learning approaches to more NLP problems in academia and industry, and encourage discussion amongst experts and practitioners from the two realms
- Motivate works on comparing different meta-learning methods and comparing meta-learning to other transfer learning methods that have been long utilized for low-resource NLP
- Encourage communication within the field of NLP to share knowledge, ideas, and data for meta-learning, and encourage future collaboration to inspire innovation.
Call for Papers
MetaNLP workshop invites submissions that investigate the theoretical and experimental nature of meta learning methodologies and their applications to NLP. Relevant research directions include, but not limited to:
- New meta-learning approaches
- Application of meta-learning models to NLP tasks, such as parsing, dialog system, question answering, summarization, translation
- Generalizability of meta-learned models across domains, tasks, or languages
- Understanding of why do meta-learning methods work for NLP, for example:
- What does the model learn in meta-learning tasks?
- Are there some meta-learning approaches that are suitable for some NLP applications but not others?
- Comparative study on meta-learning approaches.
Popular meta-learning topics include, but not limited to:
- Learning optimizer
- Learning model initialization
- Learning metrics or distance measurement
- Learning training algorithm
- Few shot learning
- Network architecture search
We welcome three categories of papers: regular workshop papers, cross-submissions, and extended abstracts. Only the regular workshop paper will be included in the proceedings. The extended Abstracts and cross-submissions will simply be hosted on our websites. Submissions should be made to softconf.
- Regular Workshop Papers The submissions should be in ACL 2021 style between 4 and 8 pages, excluding the references. Authors can add supplementary material in addition to the 8 pages, but reviewers are not required to review the extra material. The papers should present novel research. The review will be double blind and thus all submissions should be anonymized. Double submission is allowed, but the paper accepted by another conference should be moved to cross-submissions.
- Extended Abstracts Preliminary but interesting ideas that have not been published before can be submitted as extended abstracts. Ideas and works that would benefit from additional exposure and discussion but are not ready for publication are welcome to be submitted. The submissions should be up to 2 pages long including the references. The review will be double blind and thus all submissions should be anonymized.
- Cross-Submissions We also invite works on relevant topics that have appeared in or submitted to alternative venues. Accepted cross-submissions will be presented as posters, with an indication of the original venue. Selection of cross-submissions will be determined solely by the organizing committee.
- Accepted papers for all the three tracks will get one additional page to address reviewer comments.
Important Dates
- Paper Submissions Due:
April 26May 7, 2021 (AoE) - Notification of Acceptance:
May 28May 31, 2021 (AoE) - Camera-ready Paper Due: June 7, 2021 (AoE)
- Workshop Date: August 5, 2021
Invited Speakers
Andreas Vlachos
University of Cambridge
Chelsea Finn
Stanford University
Eric Xing
Carnegie Mellon University
Heng Ji
University of Illinois Urbana-Champaign
Zhou Yu
Columbia University
Program
Schedule (EDT)
6:00-6:15 Opening remarks
6:15-7:00 Invited talk - Meta-Learning for few-shot learning in NLP - Andreas Vlachos
7:00-7:20 Contributed talk - Don't Miss the Labels: Label-semantic Augmented Meta-Learner for Few-Shot Text Classification
7:20-7:40 Contributed talk - Learning to Bridge Metric Spaces: Few-shot Joint Learning of Intent Detection and Slot Filling
7:40-8:00 Contributed talk - Meta-Reinforcement Learning for Mastering Multiple Skills and Generalizing across Environments in Text-based Games
8:00-8:20 Contributed talk - Few-Shot Event Detection with Prototypical Amortized Conditional Random Field
8:20-8:40 Contributed talk - Meta-Learning for Improving Rare Word Recognition in end-to-end ASR
8:40-9:00 Contributed talk - Minimax and Neyman–Pearson Meta-Learning for Outlier Languages
9:00-9:15 Coffee break
9:15-10:00 Invited talk - Meta-learning for dialog systems - Zhou Yu
10:00-10:45 Invited talk - Learning-to-learn through Model-based Optimization: HPO, NAS, and Distributed Systems - Eric Xing
10:45-11:00 Coffee break
11:00-11:20 Contributed talk - Soft Layer Selection with Meta-Learning for Zero-Shot Cross-Lingual Transfer
11:20-11:40 Contributed talk - Zero-Shot Compositional Concept Learning
11:40-12:00 Contributed talk - Few Shot Dialogue State Tracking using Meta-learning
12:00-13:00 Poster session
13:00-13:15 Coffee break
13:15-14:00 Invited talk - Few-Shot Learning to Give Feedback in the Real World - Chelsea Finn
14:00-14:45 Invited talk - Learning from Annotation Guideline: A case study on Event Extraction - Heng Ji
14:45-15:00 Closing remarks
Accepted Papers - Talk
Soft Layer Selection with Meta-Learning for Zero-Shot Cross-Lingual Transfer
Weijia Xu, Batool Haider, Jason Krone and Saab Mansour
Meta-Reinforcement Learning for Mastering Multiple Skills and Generalizing across Environments in Text-based Games
Zhenjie Zhao, Mingfei Sun and Xiaojuan Ma
Zero-Shot Compositional Concept Learning
Guangyue Xu, Parisa Kordjamshidi and Joyce Chai
Cross submissions/presentations - Talk
Meta-Learning for Improving Rare Word Recognition in end-to-end ASR [ICASSP 2021]
Florian Lux and Ngoc Thang Vu
Few Shot Dialogue State Tracking using Meta-learning [EACL 2021]
Saket Dingliwal, Shuyang Gao, Sanchit Agarwal, Chien-Wei Lin, Tagyoung Chung and Dilek Hakkani-Tur
Few-Shot Event Detection with Prototypical Amortized Conditional Random Field [ACL 2021 findings]
Xin Cong, Shiyao Cui, Bowen Yu, Tingwen Liu, Wang Yubin, and Bin Wang
Minimax and Neyman–Pearson Meta-Learning for Outlier Languages [ACL 2021 findings]
Edoardo Maria Ponti, Rahul Aralikatte, Disha Shrivastava, Siva Reddy and Anders Søgaard
Don't Miss the Labels: Label-semantic Augmented Meta-Learner for Few-Shot Text Classification [ACL 2021 findings]
Qiaoyang Luo, Lingqiao Liu, Yuhao Lin, and Wei Emma Zhang
Learning to Bridge Metric Spaces: Few-shot Joint Learning of Intent Detection and Slot Filling [ACL 2021 findings]
Yutai Hou, Yongkui Lai, Cheng Chen, Wanxiang Che, and Ting Liu
Accepted Papers - Posters
Multi-Pair Text Style Transfer for Unbalanced Data via Task-Adaptive Meta-Learning
Xing Han and Jessica Lundin
On the cross-lingual transferability of multilingual prototypical models across NLU tasks
Oralie Cattan, Sophie Rosset and Christophe Servan
Meta-Learning for Few-Shot Named Entity Recognition
Cyprien de Lichy, Hadrien Glaude and William Campbell
Multi-accent Speech Separation with One Shot Learning
Kuan Po Huang, Yuan-Kuei Wu and Hung-yi Lee
Semi-supervised Meta-learning for Cross-domain Few-shot Intent Classification
Yue Li and Jiong Zhang
Meta-learning for Classifying Previously Unseen Data Source into Previously Unseen Emotional Categories
Gaël Guibon, Matthieu Labeau, Hélène Flamein, Luce Lefeuvre, and Chloé Clavel
Accepted extended abstract - Posters
Meta-learning for Task-oriented Household Text Games
Zhenjie Zhao and Xiaojuan Ma
Meta-learning for downstream aware and agnostic pretraining
Hongyin Luo, Shuyan Dong, Yung-Sung Chuang and Shang-Wen Li
Organizers
Hung-Yi Lee
Associate Professor, National Taiwan University
Mitra Mohtarami
Research Scientist, Massachusetts Institute of Technology
Shang-Wen Li
Senior Applied Scientist, Amzaon web services AI
Di Jin
Applied Scientist, Amazon Alexa AI
Mandy Korpusik
Assistant Professor, Loyola Marymount University
Annie Dong
Applied Scientist, Amazon Alexa AI
Ngoc Thang Vu
Professor, University of Stuttgart
Dilek Hakkani-Tur
Senior Principal Scientist, Amazon Alexa AI
Program committee
- Trapit Bansal (University of Massachusetts, Amherst)
- Yangbin Chen (City University of Hong Kong)
- Yutian Chen (DeepMind)
- Samuel Coope (PolyAI)
- Jennifer Drexler (Rev)
- Tianyu Gao (Princeton University)
- Xavi Gonzalvo (Google Research)
- Yutai Hou (Harbin Institute of Technology)
- Kuan-Po Huang (National Taiwan University)
- Sathish Reddy Indurthi (Samsung)
- Ankit Jain (Facebook)
- Ping Jian (Beijing Institute of Technology)
- Tom Ko (Southern University of Science and Technology)
- Cheng-I Lai (Massachusetts Institute of Technology)
- Zhaojiang Lin (The Hong Kong University of Science and Technology)
- Yijia Liu (Alibaba Group)
- Colin Lockard (Amazon)
- Hongyin Luo (Massachusetts Institute of Technology)
- Meryem M'hamdi (University of Southern California)
- Andrea Madotto (The Hong Kong University of Science and Technology)
- Fei Mi (École Polytechnique Fédérale de Lausanne)
- Mehrad Moradshahi (Stanford University)
- Hoang Long Nguyen (Apple)
- Mohammad Salameh (University of Alberta)
- Hsuan Su (National Taiwan University)
- Jian Sun (Alibaba Group)
- Chenglong Wang (University of Washington)
- Yuan-Kuei Wu (National Taiwan University)
- Yu Zhang (Google Brain)
Reading
Meta learning is one of the fastest growing research areas in the deep learning scope. However there is no standard definition for meta learning. Usually the main goal is to design models that can learn new tasks rapidly with few in domain training examples, by having models to pre-learn from many, relevant or not, training tasks in a way that the models ar e easy to be generalized to new tasks. For better understanding the scope of meta learning, we provide several online courses and papers describing the works falling into the area. These works are just for showcasing, and we definitely encourage people with research not covered here but sharing the same goal mentioned above to submit.
Online Courses
Papers
Meta Learning Technology
- Learning to Initialize:
- Chelsea Finn, Pieter Abbeel, and Sergey Levine, “Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks”, ICML, 2017
- Sebastian Flennerhag, Pablo G. Moreno, Neil D. Lawrence, Andreas Damianou, Transferring Knowledge across Learning Processes, ICLR, 2019
- Learning to optimize:
- Sachin Ravi, Hugo Larochelle, Optimization as a model for few-shot learning, ICLR, 2017
- Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, Nando de Freitas, Learning to learn by gradient descent by gradient descent, NIPS, 2016
- Learning to compare
- Jake Snell, Kevin Swersky, Richard S. Zemel, Prototypical Networks for Few-shot Learning, NIPS, 2017
- Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, Daan Wierstra, Matching Networks for One Shot Learning, NIPS, 2016
- Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H.S. Torr, Timothy M. Hospedales, Learning to Compare: Relation Network for Few-Shot Learning, CVPR, 2018
- Learning the whole learning algorithm
- Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy Lillicrap, Meta-Learning with Memory-Augmented Neural Networks, ICML, 2016
- Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, Pieter Abbeel, A Simple Neural Attentive Meta-Learner, ICLR, 2018
- Network architecture search:
- RL based
- Barret Zoph, Quoc V. Le, Neural Architecture Search with Reinforcement Learning, ICLR 2017
- Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le, Learning Transferable Architectures for Scalable Image Recognition, CVPR, 2018
- Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, Jeff Dean, Efficient Neural Architecture Search via Parameter Sharing, ICML, 2018
- Evolution based
- Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, Alex Kurakin, Large-Scale Evolution of Image Classifiers, ICML 2017
- Esteban Real, Alok Aggarwal, Yanping Huang, Quoc V Le, Regularized Evolution for Image Classifier Architecture Search, AAAI, 2019
- Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, Koray Kavukcuoglu, Hierarchical Representations for Efficient Architecture Search, ICLR, 2018
- Supernetwork based
- Hanxiao Liu, Karen Simonyan, Yiming Yang, DARTS: Differentiable Architecture Search, ICLR, 2019
- Bayesian optimisation based
- Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabás Póczos, Eric P Xing, Neural Architecture Search with Bayesian Optimisation and Optimal Transport, NeurIPS 2018
- Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger, Biswajit Paria, Christopher R. Collins, Jeff Schneider, Barnabas Poczos, Eric P. Xing, Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly, JMLR, 2020
- RL based