| CARVIEW |
The First Workshop on the Evaluation of Generative Foundation Models at CVPR 2024
Workshop Time, June 18 2024, 1pm - 6:30pm PDT
Workshop location, Seattle Convention Center | Summit 433
The landscape of artificial intelligence is being transformed by the advent of Generative Foundation Models (GenFMs), such as Large Language Models (LLMs) and diffusion models. GenFMs offer unprecedented opportunities to enrich human lives and transform industries. However, they also pose significant challenges, including the generation of factually incorrect or biased information, which might be potentially harmful or misleading. With the emergence of multimodal GenFMs, which leverage and generate content in an increasing number of modalities, these challenges are set to become even more complex. This emphasizes the urgent need for rigorous and effective evaluation methodologies.
The 1st Workshop on Evaluation for Generative Foundation Models at CVPR 2024 aims to build a forum to discuss ongoing efforts in industry and academia, share best practices, and engage the community in working towards more reliable and scalable approaches for GenFMs evaluation.
Workshop Schedule
Gathering + Posters Arrangement
Jun 18, 13:00 - 13:20 PDTOpening Remarks
Jun 18, 13:20 - 13:30 PDTKeynote, Ludwig Schmidt
DataComp: Evaluating Training Sets for Multimodal Models
Jun 18, 13:30 - 14:00 PDT
Keynote, Ranjay Krishna
The past, Present, and Future of Vision-Language Evaluation
Abstract Jun 18, 14:00 - 14:30 PDT
Keynote, Hanwang Zhang
Road to Level 5 MM Generalist
Abstract Jun 18, 14:30 - 15:00 PDT
Spotlight Presentations
Jun 18, 15:00 - 15:15 PDTPoster Session, Coffee Break
Jun 18, 15:15 - 15:45 PDTKeynote, Leonid Karlinsky
Analyzing and Improving Compositional Reasoning in Multi-Modal Foundation Models
Abstract Jun 18, 15:45 - 16:15 PDT
Oral Presentation, Linjie Li
Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation
Abstract Jun 18, 16:15 - 16:30 PDT
Panel Discussion
Jun 18, 16:30 - 17:00 PDTKeynote, Sadeep Jayasumana
Rethinking FID: Towards a Better Evaluation Metric for Image Generation
Abstract Jun 18, 17:00 - 17:25 PDT
Keynote, Jungo Kasai
Dramatic Five Years of AI and NLP: Evaluation and the Future of Foundation Models
Abstract Jun 18, 17:25 - 17:50 PDT
Keynote, Bo Li
Risk Assessment, Safety Alignment, and Guardrails for Generative Models
Abstract Jun 18, 17:50 - 18:15 PDT
Closing Remarks
Jun 18, 18:15 - 18:30 PDT
Invited Speakers and Panelists
Ludwig Schmidt
Assistant Professor at University of Washington
Bo Li
Associate Professor at University of Chicago
Ranjay Krishna
Associate Professor at University of Washington
Hanwang Zhang
Associate Professor at Nanyang Technological University
Leonid Karlinsky
Principal Research Scientist at IBM Research
Jungo Kasai
Co-founder & CTO at Kotoba Technologies, Inc.
Sadeep Jayasumana
Staff Research Scientist at Google Research
Besmira Nushi
Researcher at Microsoft Research
Lucas Beyer
Staff Research Engineer at Google Brain
Accepted Papers
- Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation; Jaemin Cho, Linjie Li, Zhengyuan Yang, Zhe Gan, Lijuan Wang, Mohit Bansal.
- Evaluating and Improving Compositional Text-to-Visual Generation; Baiqi Li, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Xide Xia, Graham Neubig, Pengchuan Zhang, Zhiqiu Lin, Deva Ramanan.
- Evaluating Multimodal Large Language Models across Distribution Shifts and Augmentations; Aayush Atul Verma, Amir Saeidi, Shamanthak Hegde, Ajay Therala, Fenil Denish Bardoliya, Nagaraju Machavarapu, Shri Ajay Kumar Ravindhiran, Srija Malyala, Agneet Chatterjee, Yezhou Yang, Chitta Baral.
- Towards Quantitative Evaluation Metrics for Image Editing Approaches; Dana Cohen Hochberg, Oron Anschel, Alon Shoshan, Igor Kviatkovsky, Manoj Aggarwal, Gerard G Medioni.
- ReMOVE: A Reference-free Metric for Object Erasure; Aditya Chandrasekar, Goirik Chakrabarty, Jai Bardhan, Ramya Hebbalaguppe, Prathosh A P.
Program Committee
Alex C Williams
Amazon
Amir Tavanaei
Amazon
Angel Martinez-Gonzalez
Amazon
Bardiya Akhbari
Amazon
Ebrahim Safadi
Amazon
Fatemeh Ghezloo
University of Washington
Gauthier Guinet
Amazon
Gokhan Kirlik
Amazon
Gozde Sahin
Amazon
Hammad Ayyubi
Columbia University
Hengameh M Dastjerdi
Amazon
Hengkang Wang
University of Minnesota
Huitian Lei
Lyft INC
Jiawei Ma
Columbia University
Junyi Hu
UMass at Lowell
Kee Kiat Koo
Amazon
Mahtab Bigverdi
University of Washington
Meng Han
Amazon
Michael Opitz
Amazon
Nadine Behrmann
Bosch Center for AI
Nicholas Dronen
Amazon
Pradeep Yarlagadda
Amazon
Prateek Singhi
Amazon
Qi Li
Amazon
Rahul Sharma
Amazon
Rustin Soraki
University of Washington
Thomas Cilloni
Amazon
Tristan McKinney
Amazon
Wentai Zhang
Amazon
Wenyi Wu
Amazon
Wisdom Ikezogwo
University of Washington
Xiao Zhang
Amazon
Xin Li
Amazon
Yiming Qian
Amazon
Yuguang Li
Zillow Group
Special thanks: we thank Ece Kamar from Microsoft Research for her support of our workshop.