Research Projects

CLiMB: The Continual Learning in Multimodality Benchmark

Collaborators: Ting-Yun Chang, Leticia Pinto Alva
Advisors: Jesse Thomason, Mohammad Rostami

CLiMB is a benchmark to study the novel challenge of learning vision-and-language tasks in a continual learning setting.

NeurIPS Datasets and Benchmarks 2022 Code Slides Video

Gender Bias in Pre-Trained Vision-and-Language Models

Advisor : Yonatan Bisk

We analyze intra- and inter-modality gender biases encoded by pre-trained vision-and-language models, which often prefer to reinforce stereotypes over faithfully describing the visual scene.

GeB4NLP 2022

Multimodal ASR for Recovering Noisy and Corrupted Speech

Collaborator : Ramon Sanabria
Advisors: Desmond Elliott, Florian Metze

We investigate the utility of multimodal ASR under noisy conditions, showing that the visual context can be leveraged to recover masked words in the speech signal.

ICASSP 2020 Findings of EMNLP 2020 NLPBT 2020 Code

Tejas Srinivasan

CLiMB: The Continual Learning in Multimodality Benchmark

Gender Bias in Pre-Trained Vision-and-Language Models

Multimodal ASR for Recovering Noisy and Corrupted Speech