This repo contains the code for the CVPR 2018 paper "Semantic Video Segmentation by Gated Recurrent Flow Propagation" by David Nilsson and Cristian Sminchisescu. [pdf]

Check config.py. Download all data from the cityscapes dataset and change the paths in config.py. Check that you can run python config.py without any errors.

Run misc/compile.sh to compile the bilinear warping operator. Change the include directory on line 9 if you get errors related to libcudart.

Download all pretrained models from here and unpack them under ./checkpoints/. For instance, the file ./checkpoints/flownet1.index should exist.

Evaluate the GRFP(LRR-4x, FlowNet2) setup on the validation set by running:

python evaluate.py --static lrr --flow flownet2

Evalutate GRFP(Dilation10, FlowNet2) for various number of frames, as in Table 3 and 4 in the paper:

python evaluate.py --static dilation --flow flownet2 --frames 1
python evaluate.py --static dilation --flow flownet2 --frames 5

The values in table 9 can be reproduced by running the following:

python evaluate.py --static lrr --flow flownet2
python evaluate.py --static lrr --flow flownet1
python evaluate.py --static lrr --flow farneback

Train and evaluate a model with the following commands:

python train.py --static lrr --flow flownet2
python evaluate.py --static lrr --flow flownet2 --ckpt lrr_flownet2_it10000

This should match the performance of the pre-trained LRR model above. See the ./checkpoints directory where parameters are saved during the training procedure. Only LRR is supported at the moment.

If you use the code in your own research, please cite

@InProceedings{Nilsson_2018_CVPR,
author = {Nilsson, David and Sminchisescu, Cristian},
title = {Semantic Video Segmentation by Gated Recurrent Flow Propagation},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2018}
}

Depending on the setup you use, consider also citing PSP, LRR, Dilation, FlowNet1, FlowNet2 or Farnebäck.