This is the implementation of the paper "End-to-end weakly-supervised semantic alignment" by I. Rocco, R. Arandjelović and J. Sivic.

For more information check out the project [website] and the paper on [arXiv].

The code is implemented using Python 3 and PyTorch 0.2. All dependencies are included in the standard Anaconda distribution.

The code includes scripts for pre-training the models with strong supervision (train_strong.py) as proposed in our previous work, as well as to fine-tune the model using weak supervision (train_weak.py) as proposed in this work.

Training scripts can be found in the scripts/ folder.

Evaluation is implemented in the eval.py file. It can evaluate a single affine or TPS model (with the --model-aff and --model-tps parameters respectively), or a combined affine+TPS model (with the --model) parameter.

The evaluation dataset is passed with the --eval-dataset parameter.

Trained models for the baseline method using only strong supervision and the proposed method using additional weak supervision are provided below. You can store them in the trained_models/ folder.

With the provided code below you should obtain the results from Table 2 of the paper.

CNNGeometric with VGG-16 baseline: [affine model],[TPS model]

python eval.py --feature-extraction-cnn vgg --model-aff trained_models/cnngeo_vgg16_affine.pth.tar --model-tps trained_models/cnngeo_vgg16_tps.pth.tar --eval-dataset pf-pascal

CNNGeometric with ResNet-101 baseline: [affine model],[TPS model]

python eval.py --feature-extraction-cnn resnet101 --model-aff trained_models/cnngeo_resnet101_affine.pth.tar --model-tps trained_models/cnngeo_resnet101_tps.pth.tar --eval-dataset pf-pascal

Proposed method: [combined aff+TPS model]

python eval.py --feature-extraction-cnn resnet101 --model trained_models/weakalign_resnet101_affine_tps.pth.tar --eval-dataset pf-pascal

If you use this code in your project, please cite our paper:

@article{Rocco18,
        author       = "Rocco, I. and Arandjelovi\'c, R. and Sivic, J.",
        title        = "End-to-end weakly-supervised semantic alignment",
        journal={arXiv preprint arXiv:1712.06861},
         }