Modules for performing class-importance-weighted domain alignment (IWDA) in PyTorch for unsupervised domain adaptation, as well as on text and token classifiers built upon pre-trained language models using the 🤗 Transformers library. The alignment is implemented using adversarial training with Wasserstein-1 critic loss and zero-centered gradient penalty.

The modules are found in domain_alignment.py. They are W1CriticWithImWeights, W1CriticWithImWeightsEstimation, and ImWeightsEstimator. See run_text_cls.py, run_token_cls.py and below for example usage.

One usage of IWDA is unsupervised cross-lingual transfer of pre-trained language models on downstream tasks, discussed and evaluated in our ICLR 2022 paper “Cross-Lingual Transfer with Class-Weighted Language-Invariant Representations”.

The commands for the following unsupervised cross-lingual transfer tasks are provided in run_iclr2022.sh (also check package requirements.txt):

• mBERT transfer from English to Spanish for named-entity recognition on CoNLL-2002 and 2003 datasets.
• mBERT transfer from English to Japanese for sentiment analysis on Multilingual Amazon Reviews Corpus.

Results on CoNLL NER with mBERT (average of 5 runs):

Results on MARC sentiment analysis with mBERT (average of 3 runs):

The BibTeX entry for our paper is:

@inproceedings{xian2022crosslingual,
  title={Cross-Lingual Transfer with Class-Weighted Language-Invariant Representations},
  author={Ruicheng Xian and Heng Ji and Han Zhao},
  year={2022},
  booktitle={International Conference on Learning Representations},
  url={https://openreview.net/forum?id=k7-s5HSSPE5}
}

Our implementation follows the following paper closely, where some code come from:

@inproceedings{tachetdescombes2020domainadaptation,
  title={Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift},
  author={Tachet des Combes, Remi and Zhao, Han and Wang, Yu-Xiang and Gordon, Geoff},
  year={2020},
  booktitle={Advances in Neural Information Processing Systems}
}