Introducing Seaweed

Seaweed, short for "Seed-Video," is a research effort for building a foundational model for video generation. This webpage showcases diffusion transformers with approximately 7 billion (7B) parameters, trained using compute equivalent to 1,000 H100 GPUs. Seaweed learns world representation from massive amounts of multi-modal data such as video, image, and text. It allows for creating videos of various resolutions, aspect ratios, and durations from text descriptions. In this article, we present its generated videos and highlight its hallmark capability as a foundational model capable of supporting a wide range of downstream applications.

Our model is highly adept at generating lifelike human characters that exhibit a diverse array of actions, gestures, and emotions.

Seaweed excels at generating a wide variety of landscapes. With intricate detail and dynamic composition, it can create visually stunning environments that enhance storytelling.

We demonstrate our model's generative capabilities through a short film. All the videos are generated and the only manually added components are the background music and the ending titles.

Generate videos from images

Our video generation model offers enhanced controls that allow users to precisely create the content they envision. By providing an image as the first frame, users can direct the model to generate the rest of the video with consistent motion and style. This grants users full control over the visual aesthetics, making it ideal for applications where accuracy and creative direction are crucial.

Our model can also condition on both the first and last frames, allowing it to generate interesting transition videos for greater creative control.

Generate videos by references

Our model can also be finetuned to generate videos based on reference images, offering flexible input options for users. Whether it's a human reference image, an object reference image, or a combination of multiple reference images, the model can synthesize them into dynamic video sequences.

Human-centric video generation

Seaweed is adapted to generate content conditioned on audio inputs by Omnihuman, enabling the creation of realistic human characters that perfectly match the voice in the audio. The model ensures synchronized lip movements and body gestures that align with the tone and timing of the audio, creating a seamless and lifelike interaction.

Generate audio with video

Seaweed is also capable of generating both audio and video together. The audio generated is synced to reflect the action, scene, tone, rhythm, and style of the video. The audio complements and elevates the visual storytelling, providing a seamless multimedia experience.

Real-time generation

Seaweed can also generate videos in real-time at 1280x720 resolution and 24fps. This is particularly valuable for real-time and interactive applications, where immediate video generation is essential.

World exploration

Seaweed can be utilized for modeling precise camera control through defined trajectories, providing not only enhanced creative direction but also an interactive way for users to explore the simulated world. With its real-time generation capability, Seaweed also serves as a foundational model for advanced research in world simulation.

Enhanced Physically-Consistent Generation

Seaweed can also be post-trained on synthetic video rendered via computer-generated imagery (CGI), enabling it to enhance the physical consistency in video generation while preserving photorealism. Below, we showcase generated videos with superior 3D consistency and precise human pose integrity in complex actions, alongside the synthetic videos used for training.