AuroraCap

AuroraCap: A Efficient and Performant Video Detailed Captioner

Architecture

LLaVA. To effectively leverage the capabilities of both the pre-trained LLM and visual model, LLaVA adapt a simple multilayer perceptron (MLP) projection layer to connect each patch tokens of image features into the word embedding space.

Token merging. To increase the throughput of existing ViT models, Token Merging is proposed to gradually combines similar tokens in a transformer to reduce the number of tokens passing through ViT models. Token Merging has been proven to be effective on image and video classification tasks even without the need for training. We conduct frame-wise token merging in AuroraCap, where the feature is extracted by CLIP ViT-H model. We show token merging visualization examples from COCO, VG, SA-1B as follows:

Training Recipe

We use over 20 million high-quality image/video-text pairs to train AuroraCap in three stages. The training datasets are released at HuggingFace.

Pretraining stage. We first align visual features with the word embedding space of LLMs. To achieve this, we freeze the pretrained ViT and LLM, training solely the vision-language connector.

Vision stage. We unfreeze the pretrained ViT while freezing the LLM during vision stage and train with the public data among various computer vision tasks to get better generalization.

Language stage. Finally, we conduct end-to-end training, which means all the components are trainable, with the most high-quality public data during language stage.

VDC: A New Video Detailed Captioning Benchmark

Benchmark Collection and Processing

Video collection and processing. We building VDC upon Panda-70M, Ego4D, Mixkit, Pixabay, and Pexels. We first split the video into clips and apply dense frame extraction, then manually replacing blurry frames with adjacent clear ones.

Structured detailed captions construction pipeline. We develop a structured detailed captions construction pipeline to generate extra detailed descriptions from various perspectives, significantly extending the length and enhancing the richness compared to previous benchmarks. The structured detailed captions includes the following categories:

1. Camera caption. Describe the camera work in detail, including shot types, angles, movements, transitions, and any special effects used to enhance the video.
2. Short caption. Summarize the video in one detailed sentence, capturing key actions and the overall mood.
3. Background caption. Provide a detailed description of the background, including objects, location, weather, time, and any dynamic elements.
4. Main Object caption. Give a thorough description of the main subject's actions, attributes, interactions, and movements throughout the video frames.
5. Detailed caption. Generate a detailed, vivid caption for the video, covering all categories, ensuring it's engaging, informative, and rich enough for AI to recreate the video content.

To generate detailed, fine-grained, and accurate captions, we leverage GPT-4o to produce video descriptions. We design a hierarchical prompt strategy to efficiently obtain accurate structured captions and detailed captions in two conversation rounds: (1) Structured Captions Generation and (2) Detailed Captions Integration.

Evaluation Metric Design and Leaderboard

We introduce VDCscore, a novel quantitative metric that utilizes LLMs to evaluate the similarity between predicted and ground-truth detailed captions through a divide-and-conquer approach. The core idea of VDCscore is to decompose long detailed captions into multiple short question-answering pairs, avergae the evaluation of each pair as the final result.

Benchmark Examples

Evaluation

Benchmarking video detailed captioning. AuroraCap achieves superior performance in video detailed captioning while utilizing significantly fewer visual tokens than other models, fully highlighting the efficiency of AuroraCap.

We evaluate AuroraCap using CIDEr, BELU-4, BELU-1, METEOR, and ROUGE-L metric on Flickr, NoCaps, and COCO-Cap benchmarks and compare it with LLM-based state-of-the-art methods. AuroraCap shows good performance under zero-shot settings. Notice that these benchmarks all contain short captions consisting of a single sentence, so they only partially reflect the model's performance.

Video captioning.

Although the current video captioning benchmarks are only contains one-sentence captions, to compare with prior work, we similarly evaluate on these benchmarks. We evaluate AuroraCap on MSR-VTT and VATEX and compare it with other methods.

Ablation Study

As a core training and inference strategy of AuroraCap, token merging plays a significant role in reducing the number of visual tokens. We further study how the video detailed captioning capability is influenced by token merge ratio.

We define the performance percentage as the proportion between the highest and lowest values on the entire performance curve. We highlight the token merging ratio when achieving 90% and 80% performance with the dash line and filled area. We found that token merging significantly reduces the number of tokens while maintaining minimal performance drop, and even showing improvement in some tasks.

To assess the inference speed, we utilize the inference time per video question-answering pair in seconds (TPV) as an evaluative metric. Figure below indicates the minimum TPV achievable in our settings including with or without token merging and SGLang across seven video understanding datasets. Reducing the visual tokens and using SGLang result in excellent inference times per video question-answering pair while all the datasets with short video and question inputs.

Case Study

We perform an extensive case study of AuroraCap on a variety of videos for video detailed captioning. As shown as followings, AuroraCap is capable of providing excellent detailed captions regarding the camera motion, background and main object with less hallucination.