HTTP/2 200 server: GitHub.com content-type: text/html; charset=utf-8 last-modified: Tue, 30 Sep 2025 18:56:43 GMT access-control-allow-origin: * etag: W/"68dc27eb-cc69" expires: Sun, 28 Dec 2025 19:10:28 GMT cache-control: max-age=600 content-encoding: gzip x-proxy-cache: MISS x-github-request-id: E271:2DDCFF:7EBFF2:8E24A0:69517E4C accept-ranges: bytes age: 0 date: Sun, 28 Dec 2025 19:00:28 GMT via: 1.1 varnish x-served-by: cache-bom-vanm7210052-BOM x-cache: MISS x-cache-hits: 0 x-timer: S1766948428.180072,VS0,VE204 vary: Accept-Encoding x-fastly-request-id: 25276dd06c4621f539044613fa7238390ad394c2 content-length: 10118

Idan Schwartz

Assistant Professor

Bar-Ilan University

Research & Bio

I'm an Assistant Professor at Bar-Ilan University, leading the Multimodal Lab.

My research focuses on multimodal problems, primarily generative, with challenges including modal alignment and efficient inference-time solutions.

I also have a strong interest in connecting ideas from cognitive science decision making to deep learning concepts, such as model perceptiveness, representation as comprehension, attention, and programming as system 2 problem solving.

I completed my postdoc with Prof. Lior Wolf at Tel Aviv University. Before that, I earned my PhD in Computer Science from the Technion, under the supervision of Prof. Tamir Hazan and Prof. Alexander G. Schwing from UIUC. My thesis focused on Cognitive Models in Deep Learning. You can find my thesis here.

My experience as a researcher in industry includes work on eBay's catalog (vision and language), Microsoft's Assistant (meeting insights, transcript-based), and at Spot for cloud workload optimization platform (time-series prediction). I currently serve as Chief Scientist at Aigency.ai, where I'm helping revolutionize the internet in the age of intelligent agents.

Interests

• Artificial Intelligence & Cognition
• Attention Models
• Multimodal Learning
• Computer Vision
• Natural Language Processing

Education

• Postdoc in Computer Science, 2023

  Tel-Aviv University

• PhD in Computer Science, 2022

  Technion

• BSc in Computer Science, 2015

  Technion

Featured Publications

I. Schwartz, V. Snæbjarnarson, S. Benaim, H. Chefer, R. Cotterell, L. Wolf, S. Belongie

April 2023 ICCV'23

Discriminative Class Tokens for Text-to-Image Diffusion Models

Recent advances in text-to-image diffusion models have enabled the generation of diverse and high-quality images. However, generated images often fall short of depicting subtle details and are susceptible to errors due to ambiguity in the input text. One way of alleviating these issues is to train diffusion models on class-labeled datasets. This comes with a downside, doing so limits their expressive power: (i) supervised datasets are generally small compared to large-scale scraped text-image datasets on which text-to-image models are trained, and so the quality and diversity of generated images are severely affected, or (ii) the input is a hard-coded label, as opposed to free-form text, which limits the control over the generated images. In this work, we propose a non-invasive fine-tuning technique that capitalizes on the expressive potential of free-form text while achieving high accuracy through discriminative signals from a pretrained classifier, which guides the generation. This is done by iteratively modifying the embedding of a single input token of a text-to-image diffusion model, using the classifier, by steering generated images toward a given target class. Our method is fast compared to prior fine-tuning methods and does not require a collection of in-class images or retraining of a noise-tolerant classifier. We evaluate our method extensively, showing that the generated images are: (i) more accurate and of higher quality than standard diffusion models, (ii) can be used to augment training data in a low-resource setting, and (iii) reveal information about the data used to train the guiding classifier.

PDF Cite Code

H. Chefer, I. Schwartz, L. Wolf

July 2022 Neruips'22

Optimizing Relevance Maps of Vision Transformers Improves Robustness

It has been observed that visual classification models often rely mostly on spurious cues such as the image background, which hurts their robustness to distribution changes. To alleviate this shortcoming, we propose to monitor the model’s relevancy signal and direct the model to base its prediction on the foreground object.This is done as a finetuning step, involving relatively few samples consisting of pairs of images and their associated foreground masks. Specifically, we encourage the model’s relevancy map (i) to assign lower relevance to background regions, (ii) to consider as much information as possible from the foreground, and (iii) we encourage the decisions to have high confidence. When applied to Vision Transformer (ViT) models, a marked improvement in robustness to domain-shifts is observed. Moreover, the foreground masks can be obtained automatically, from a self-supervised variant of the ViT model itself; therefore no additional supervision is required.

PDF Cite Code

Y. Tewel, Y. Shalev, I. Schwartz, L. Wolf

June 2022 CVPR'22

ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic

Recent text-to-image matching models apply contrastive learning to large corpora of uncurated pairs of images and sentences. While such models can provide a powerful score for matching and subsequent zero-shot tasks, they are not capable of generating caption given an image. In this work, we repurpose such models to generate a descriptive text given an image at inference time, without any further training or tuning step. This is done by combining the visual-semantic model with a large language model, benefiting from the knowledge in both web-scale models. The resulting captions are much less restrictive than those obtained by supervised captioning methods. Moreover, as a zero-shot learning method, it is extremely flexible and we demonstrate its ability to perform image arithmetic in which the inputs can be either images or text and the output is a sentence. This enables novel high-level vision capabilities such as comparing two images or solving visual analogy tests.

PDF Cite Code

I. Schwartz, A. ~G. Schwing, T. Hazan

June 2019 CVPR'19

Factor Graph Attention

Dialog is an effective way to exchange information, but subtle details and nuances are extremely important. While significant progress has paved a path to address visual dialog with algorithms, details and nuances remain a challenge. Attention mechanisms have demonstrated compelling results to extract details in visual question answering and also provide a convincing framework for visual dialog due to their interpretability and effectiveness. However, the many data utilities that accompany visual dialog challenge existing attention techniques. We address this issue and develop a general attention mechanism for visual dialog which operates on any number of data utilities. To this end, we design a factor graph based attention mechanism which combines any number of utility representations. We illustrate the applicability of the proposed approach on the challenging and recently introduced VisDial datasets, outperforming recent state-of-the-art methods by 1.1% for VisDial0.9 and by 2% for VisDial1.0 on MRR. Our ensemble model improved the MRR score on VisDial1.0 by more than 6%.

PDF Cite Code

Publications

Yair Shpitzer, Gal Chechik, Idan Schwartz (2025). Single Image Iterative Subject-driven Generation and Editing. arXiv.

PDF Project Page

Oz Zafar, Yuval Cohen, Lior Wolf, Idan Schwartz (2024). Detection-Driven Object Count Optimization for Text-to-Image Diffusion Models. arXiv.

PDF Project Page

G. Yariv, I. Gat, S. Benaim, L. Wolf, I. Schwartz, Y. Adi (2023). Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation . AAAI'24.

PDF Cite Code

Y. Tewel, Y. Shalev, R. Nadler, I. Schwartz, L. Wolf (2023). Zero-shot video captioning with evolving pseudo-tokens. BMVC'23.

PDF Cite Code

G. Yariv, I. Gat, S. Benaim, L. Wolf, I. Schwartz (2023). AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation. INTERSPEECH'23.

PDF Cite Code

H. Chefer, I. Schwartz, L. Wolf (2022). Optimizing Relevance Maps of Vision Transformers Improves Robustness. Neruips'22.

PDF Cite Code

Y. Tewel, Y. Shalev, R. Nadler, I. Schwartz, L. Wolf (2023). ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic CVPR'22.

PDF Cite Code

A. Ali, I. Schwartz, T. Hazan, L. Wolf (2022). Video and Text Matching with Conditioned Embeddings. WACV'22.

PDF Cite Code

T. Braude, I. Schwartz, A. ~G. Schwing, A. Shamir (2022). Ordered attention for coherent visual storytelling. ACM-MM'22.

PDF Cite

I. Gat, I. Schwartz, A. ~G. Schwing (2021). Perceptual Score: Measuring Perceptiveness of Multi-Modal Classifiers. NIPS'21.

PDF Cite Code

I. Schwartz (2021). Ensemble of MRR and NDCG models for Visual Dialog. NAACL'21.

PDF Cite Code

I. Gat, I. Schwartz, A. ~G. Schwing (2020). Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies. NeurIPS'20.

PDF Cite Code

I. Schwartz, A. ~G. Schwing, T. Hazan (2019). A Simple Baseline for Audio-Visual Scene-Aware Dialog. CVPR'19.

PDF Cite Code

I. Schwartz, A. G. Schwing, Tamir Hazan (2021). High-Order Attention Models for Visual Question Answering. NIPS'21.

PDF Cite Code

Talks

2025

Discriminative models can make generative models better (in Hebrew)

2024

Multimodal Attention, Perception, Comprehension

2019

Attention Models for Vision and Language

Contact

• idanschwartz at gmail dot com
• Computer Science Department, Room 213
  Building 503
  Bar Ilan University
  Ramat Gan, Tel Aviv District, Israel