This is the official PyTorch implementation of EcoFormer: Energy-Saving Attention with Linear Complexity by Jing Liu, Zizheng Pan, Haoyu He, Jianfei Cai, and Bohan Zhuang.

• 11/11/2022. EcoFormer is selected as Spotlight!
• 08/10/2022. We release the source code. Any issues are welcomed!
• 15/09/2022. EcoFormer is accepted by NeurIPS 2022! 🔥🔥🔥

We present a novel energy-saving attention mechanism with linear complexity, called EcoFormer, to save the vast majority of multiplications from a new binarization perspective. More details can be found in our paper.

• Python ≥ 3.8
• PyTorch 1.10.1
• CUDA 11.1
• Torchvision 0.11.2
• PyTorch Image Models (timm) 0.4.9
• MMCV 1.3.8
• Einops 0.4.1
• SciPy 1.8.0

Use Anaconda to create the running environment for the project, kindly run

git clone https://github.com/ziplab/EcoFormer
cd EcoFormer
conda env create -f environment/environment.yml
conda activate ecoformer

Note: If the above instructions does not work on your machine, please refer to environment/README.md for manual installation and trouble shootings.

For experiments on PVTv2, please refer to pvt.

For experiments on twins, please refer to twins.

If you find EcoFormer useful in your research, please consider to cite the following related papers:

@inproceedings{liu2022ecoformer,
  title={EcoFormer: Energy-Saving Attention with Linear Complexity},
  author={Liu, Jing and Pan, Zizheng and He, Haoyu and Cai, Jianfei and Zhuang, Bohan},
  booktitle={NeurIPS},
  year={2022}
}

This repository is released under the Apache 2.0 license as found in the LICENSE file.

This repository is built upon PVT and Twins. We thank the authors for their open-sourced code.