VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Models

Vision-language models (VLMs), such as CLIP, have shown strong generalization under zero-shot settings, yet adapting them to downstream tasks with limited supervision remains a significant challenge. Existing multi-modal prompt learning methods typically rely on fixed, shared prompts and deterministic parameters, which limits their ability to capture instance-level variation or model uncertainty across diverse tasks and domains.

To tackle this issue, we propose a novel Variational Multi-Modal Prompt Learning (VaMP) framework that enables sample-specific, uncertainty-aware prompt tuning in multi-modal representation learning. VaMP generates instance-conditioned prompts by sampling from a learned posterior distribution, allowing the model to personalize its behavior based on input content. To further enhance the integration of local and global semantics, we introduce a class-aware prior derived from the instance representation and class prototype. Building upon these, we formulate prompt tuning as variational inference over latent prompt representations and train the entire framework end-to-end through reparameterized sampling. Experiments on few-shot and domain generalization benchmarks show that VaMP achieves state-of-the-art performance, highlighting the benefits of modeling both uncertainty and task structure in our method.

Overview of the VaMP framework. (a) Class-Aware Prior Construction: Utilizing CLIP's frozen image encoder to process training samples, generating offline class prototypes for subsequent adaptation. (b) Variational Multi-Modal Prompt Adaptation (VMPA): Variational modeling mechanism where image-conditioned posterior and class prototype-based prior are aligned through KL divergence regularization of latent prompt distributions. (c) Training Pipeline: Full architecture of our proposed VaMP framework.

We evaluate VaMP on three challenging adaptation settings: base-to-new generalization, cross-dataset generalization, and domain generalization. VaMP consistently outperforms strong multi-modal prompt baselines while maintaining high parameter efficiency.

Base-to-Novel Generalization

Table 1: Comparison with state-of-the-art methods on base-to-novel generalization across 11 datasets.

Cross-Dataset Generalization

Table 2: Comparison with state-of-the-art methods on cross-dataset evaluation across 10 datasets.

Domain Generalization

Table 3: Comparison with state-of-the-art methods on domain generalization across 4 datasets.