A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generation

Abstract

In this work, we present CineMaster, a novel framework for 3D-aware and controllable text-to-video generation. Our goal is to empower users with comparable controllability as professional film directors: precise placement of objects within the scene, flexible manipulation of both objects and camera in 3D space, and intuitive layout control over the rendered frames. To achieve this, CineMaster operates in two stages. In the first stage, we design an interactive workflow that allows users to intuitively construct 3D-aware conditional signals by positioning object bounding boxes and defining camera movements within the 3D space. In the second stage, these control signals—comprising rendered depth maps, camera trajectories and object class labels—serve as the guidance for a text-to-video diffusion model, ensuring to generate the user-intended video content. Furthermore, to overcome the scarcity of in-the-wild datasets with 3D box and camera pose annotations, we carefully establish an automated data annotation pipeline that extracts 3D bounding boxes and camera trajectories as control signals from large-scale video data. Extensive qualitative and quantitative experiments demonstrate that CineMaster significantly outperforms existing methods and implements prominent 3D-aware text-to-video generation.

Object & Camera Motion Control Demos

Object Motion Control Demos

Camera Motion Control Demos

Qualitative Comparison

How Does it Work?

We present CineMaster, a framework that enables users to manipulate objects and camera in the 3D space for text-to-video generation. CineMaster consists of two stages. First, we present an interactive workflow that allows users to intuitively manipulate the objects and camera in a 3D-native manner. Then the control signals are rendered from the 3D engine and fed into a text-to-video diffusion model, guiding the generation of user-intended video content.

Model Design

Overview of the network architecture. We design a Semantic Layout ControlNet which consists of a semantic injector and a DiT-based ControlNet. Semantic injector fuses the 3D spatial layout and class label conditions. The DiT-based ControlNet further represents the fused features and adds to the hidden states of the base model. Meanwhile, we inject the camera trajectories by the camera adapter to achieve joint control over object motion and camera motion.

Dataset Labeling Pipeline

Dataset Labeling Pipeline. We propose a data labeling pipeline to extract 3D bounding boxes, class labels and camera poses from videos. Our pipeline consists of four steps: 1) Instance Segmentation: Obtain instance segmentation results from the foreground in videos. 2) Depth Estimation: Produce metric depth maps using DepthAnything V2. 3) 3D Point Cloud and Box Calculation: Identify the frame with the largest mask for each entity and compute the 3D point cloud of each entity through inverse projection. Then, use the minimum volume method to calculate the 3D bounding box for each entity. 4) Entity Tracking and 3D Box Adjustment: Access the point tracking results of each entity and calculate the 3D bounding boxes for each frame. Finally, project the entire 3D scene into depth maps.