Semi-Supervised Video Object Segmentation

This challenge uses the newly published VISOR annotations.
Task. Given a sub-sequence of frames with M object masks in the first frame, the goal is to segment these through the remaining frames. Other objects not present in the first frame of the sub-sequence are excluded from this benchmark. Note that any of the M objects can be occluded or out of view, and can reappear during the subsequene.

Validation Submission Phase. Between Jan and end of March, interested parties can submit their results on the validation set to the leaderboard. Successful top submissions will be invited to evaluate on the hidden Test Set in the next phase

Test Submission (Final Test). Interested participants who have already registered their interest through submitting their models to the Validation Server are now invited to evaluate their models on the test set. As we do not plan to release the first frame annotations for the test set, to avoid overfitting, you are asked to submit the trained models for inference through a Docker that will be evaluated at our end. We have now announced Instructions to evaluate on the test set. You can now contact us at any point to try your model on the test server (Open for submissions until 20th of May AOE). Winners of the challenge are only decided based on the performance on the hidden test set

EPIC-SOUNDS: Audio-Based Interaction Recognition

Task. Assign an audio-class label to a trimmed segment, indicating the class of interaction taking place in the video. This uses the EPIC-SOUNDS dataset annotations.
Training input. A set of trimmed audio segments, each annotated with one of 44 class labels.
Testing input. A set of trimmed unlabelled audio segments.
Splits. Train/Val/Test splits are available here.
Evaluation metrics. Top-1/5 accuracy for audio class, on the target test set, as well as mAC mAP and mAUC for class-balanced metrics.
Leads: Jaesung Huh (University of Oxford), Jacob Chalk (University of Bristol), Evangelos Kazkos (ex- Univ of Bristol), Dima Damen (University of Bristol) and Andrew Zisserman (University of Oxford)

EPIC-SOUNDS: Audio-Based Interaction Detection

Task. Given an untrimmed video, find all temporal segments that indicate an interaction event and assign these a label. This challenge uses the EPIC-SOUNDS dataset annotations.
Training input. A set of untrimmed videos, each video annotated with a number of, potentially overlapping, start-end times and a label out of 44 class labels.
Testing input. A set of untrimmed videos.
Splits. Train (shared with recognition) /Val (shared with recognition) /Test (distinct from recognition) splits are available here.
Evaluation metrics. average mAP, on the target test set
Leads: Jacob Chalk (University of Bristol), Jaesung Huh (University of Oxford), Evangelos Kazkos (ex- Univ of Bristol), Dima Damen (University of Bristol) and Andrew Zisserman (University of Oxford)

Action Recognition

Task. Assign a (verb, noun) label to a trimmed segment.
Training input (strong supervision). A set of trimmed action segments, each annotated with a (verb, noun) label.
Training input (weak supervision). A set of untrimmed videos, each annotated with a list of (timestamp, verb, noun) labels. Note that for each action you are given a single, roughly aligned timestamp, i.e. one timestamp located around the action. Timestamps may be located over background frames or frames belonging to another action.
Testing input. A set of trimmed unlabelled action segments.
Splits. Train and validation for training, evaluated on the test split.
Evaluation metrics. Top-1/5 accuracy for verb, noun and action (verb+noun), calculated for all segments as well as unseen participants and tail classes.

Action Detection

Task. Detect the start and the end of each action in an untrimmed video. Assign a (verb, noun) label to each detected segment.
Training input. A set of trimmed action segments, each annotated with a (verb, noun) label.
Testing input. A set of untrimmed videos. Important: You are not allowed to use the knowledge of trimmed segments in the test set when reporting for this challenge.
Splits. Train and validation for training, evaluated on the test split.
Evaluation metrics. Mean Average Precision (mAP) @ IOU 0.1 to 0.5.

Unsupervised Domain Adaptation for Action Recognition

Task. Assign a (verb, noun) label to a trimmed segment, following the Unsupervised Domain Adaptation paradigm: a labelled source domain is used for training, and the model needs to adapt to an unlabelled target domain.
Training input. A set of trimmed action segments, each annotated with a (verb, noun) label.
Testing input. A set of trimmed unlabelled action segments.
Splits. Videos recorded in 2018 (EPIC-KITCHENS-55) constitute the source domain, while videos recorded for EPIC-KITCHENS-100's extension constitute the unlabelled target domain. This challenge uses custom train/validation/test splits, which you can find here.
Evaluation metrics. Top-1/5 accuracy for verb, noun and action (verb+noun), on the target test set.

Multi-Instance Retrieval

Tasks. Video to text: given a query video segment, rank captions such that those with a higher rank are more semantically relevant to the action in the query video segment. Text to video: given a query caption, rank video segments such that those with a higher rank are more semantically relevant to the query caption.
Training input. A set of trimmed action segments, each annotated with a caption. Captions correspond to the narration in English from which the action segment was obtained.
Testing input. A set of trimmed action segments with captions. Important: You are not allowed to use the known correspondence in the Test set
Splits. This challenge has its own custom splits, available here.
Evaluation metrics. normalised Discounted Cumulative Gain (nDCG) and Mean Average Precision (mAP). You can find more details in our paper.