SL

The Short-Films 20K (SF20K) Competition aims to advance story-level video understanding by leveraging the new SF20K dataset. While recent multimodal models have demonstrated progress in video understanding, existing benchmarks are largely limited to short videos with simple narratives. In contrast, our competition focuses on complex, long-term reasoning in storytelling by introducing multiple-choice and open-ended question answering tasks.
The competition is based on SF20K-Test-Expert, a subset of the SF20K dataset, which includes manually crafted open-ended questions. The questions are designed to be challenging, requiring long-term reasoning and multimodal understanding of the video content.

Competition Format

• Main track - Unlimited model size
  This track allows for models of any size, encouraging participants to push the limits of performance in video understanding. Participate via the SF20K Competition platform.
• Special track - Constrained model size (<8B)
  This track challenges participants to create efficient models under 8 billion parameters, focusing on innovative solutions. Top scores will be required to provide reproducible code. Participate via the [CONSTRAINED MODEL SIZE] SF20K Competition platform.

Competition Phases

• Phase 1: Public Test Set Evaluation
  During this initial phase, methods are evaluated against the public test set. This phase is primarily for validation purposes, allowing participants to refine their approaches.
• Phase 2: Private Test Set Evaluation
  For the private leaderboard, we will evaluate models on new, unreleased movies to prevent data contamination. This private test set will be released on October 3rd, 2025. Final rankings will be determined based on performance in this phase.
  

Important Dates

• 01 Jul, 2025: The competition server launches with data from the public test set
• 03 Oct, 2025: Submission deadline for leaderboard ranking on the public test set
• 03-10 Oct, 2025: Private test set evaluation
• 10 Oct, 2025: Final rankings announced on the private test set
• 19 Oct, 2025: Workshop at ICCV 2025 with winner's presentations

• Starter Code
• Main Track (Unlimited Model Size)
• Special Track (Constrained Model Size)

Final Rankings

Main Track

Special Track