AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation

Learn more about AV-LINK!

We propose AV-Link, a unified framework for Video-to-Audio and Audio-to-Video generation that leverages the activations of frozen video and audio diffusion models for temporally-aligned cross-modal conditioning. The key to our framework is a Fusion Block that enables bidirectional information exchange between our backbone video and audio diffusion models through a temporally-aligned self attention operation. Unlike prior work that uses feature extractors pretrained for other tasks for the conditioning signal, AV-Link can directly leverage features obtained by the complementary modality in a single framework i.e. video features to generate audio, or audio features to generate video. We extensively evaluate our design choices and demonstrate the ability of our method to achieve synchronized and high-quality audiovisual content, showcasing its potential for applications in immersive media generation.

Compared to current Video-to-Audio and Audio-to-Video methods, AV-Link provides a unified framework for these two tasks. Rather than relying on feature extractors pretrained for other tasks (e.g. CLIP, CLAP), we directly leverage the activations from pretrained frozen Flow Matching models using a Fusion Block to achieve precise time alignment between modalities. Our approach offers competitive semantic alignment and improved temporal alignment in a self-contained framework for both modalities.

Explore AV-Link Capabilities!

Explore our model's capabilities in both Audio-to-Video and Video-to-Audio generation tasks. Use the links below to navigate to different pages showcasing our results.

Video-to-Audio Generation

• Audio Control with Text Prompt
• Comparison on In-The-Wild Examples
• Comparison on Movie Gen Benchmark

Audio-to-Video Generation

• Baseline Comparison on VGGSounds
• Text-guided Results on VGGSounds