AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark

• VDC

We present a quantitative comparison between AuroraCap with existing state-of-the-art large multimodal models across various sections of structured captions in VDC. We use LLaMA-3.1-8B as the LLM evaluation assistant. # F stands for the frame sampling number of the input video, and TPF represents the visual tokens per frame. The average key frame number in VDC is 10.

We thank Aria team, VideoChat-Flash team, LLaVAction team, Apollo team, and Cockatiel team for their contributions to the leaderboard.

Architecture

LLaVA. To effectively leverage the capabilities of both the pre-trained LLM and visual model, LLaVA adapt a simple multilayer perceptron (MLP) projection layer to connect each patch tokens of image features into the word embedding space.

Token merging. To increase the throughput of existing ViT models, Token Merging is proposed to gradually combines similar tokens in a transformer to reduce the number of tokens passing through ViT models. Token Merging has been proven to be effective on image and video classification tasks even without the need for training. We conduct frame-wise token merging in AuroraCap, where the feature is extracted by CLIP ViT-H model. We show token merging visualization examples from COCO, VG, SA-1B as follows:

Training Recipe

We use over 20 million high-quality image/video-text pairs to train AuroraCap in three stages. The training datasets are released at HuggingFace.

Pretraining stage. We first align visual features with the word embedding space of LLMs. To achieve this, we freeze the pretrained ViT and LLM, training solely the vision-language connector.

Vision stage. We unfreeze the pretrained ViT while freezing the LLM during vision stage and train with the public data among various computer vision tasks to get better generalization.

Language stage. Finally, we conduct end-to-end training, which means all the components are trainable, with the most high-quality public data during language stage.

Benchmark Collection and Processing

Video collection and processing. We building VDC upon Panda-70M, Ego4D, Mixkit, Pixabay, and Pexels. We first split the video into clips and apply dense frame extraction, then manually replacing blurry frames with adjacent clear ones.

Structured detailed captions construction pipeline. We develop a structured detailed captions construction pipeline to generate extra detailed descriptions from various perspectives, significantly extending the length and enhancing the richness compared to previous benchmarks. The structured detailed captions includes camera, short, background, main object, and detailed captions.

To generate detailed, fine-grained, and accurate captions, we leverage GPT-4o to produce video descriptions. We design a hierarchical prompt strategy to efficiently obtain accurate structured captions and detailed captions in two conversation rounds: (1) Structured Captions Generation and (2) Detailed Captions Integration.

VDCscore: Evaluating Detailed Captions with LLMs

We introduce VDCscore, a novel quantitative metric that utilizes LLMs to evaluate the similarity between predicted and ground-truth detailed captions through a divide-and-conquer approach. The core idea of VDCscore is to decompose long detailed captions into multiple short question-answering pairs, avergae the evaluation of each pair as the final result.

Benchmarking video detailed captioning.

AuroraCap achieves superior performance in video detailed captioning while utilizing significantly fewer visual tokens than other models, fully highlighting the efficiency of AuroraCap.