- **dnn_superres**: Superresolution using CNNs -- Contains four trained convolutional neural networks to upscale images.

set(the_description "Super Resolution using CNNs")

ocv_define_module(dnn_superres opencv_core opencv_imgproc opencv_dnn

OPTIONAL opencv_datasets opencv_quality # samples

)

This module contains several learning-based algorithms for upscaling an image.

Run the following command to build this module:

cmake -DOPENCV_EXTRA_MODULES_PATH=<opencv_contrib>/modules -Dopencv_dnn_superres=ON <opencv_source_dir>

Refer to the tutorials to understand how to use this module.

There are four models which are trained.

Trained models can be downloaded from [here](https://github.com/Saafke/EDSR_Tensorflow/tree/master/models).

- Size of the model: ~38.5MB. This is a quantized version, so that it can be uploaded to GitHub. (Original was 150MB.)

- This model was trained for 3 days with a batch size of 16

- Link to implementation code: https://github.com/Saafke/EDSR_Tensorflow

- x2, x3, x4 trained models available

- Advantage: Highly accurate

- Disadvantage: Slow and large filesize

- Speed: < 3 sec for every scaling factor on 256x256 images on an Intel i7-9700K CPU.

- Original paper: [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://arxiv.org/pdf/1707.02921.pdf) [1]

Trained models can be downloaded from [here](https://github.com/fannymonori/TF-ESPCN/tree/master/export).

- Size of the model: ~100kb

- This model was trained for ~100 iterations with a batch size of 32

- Link to implementation code: https://github.com/fannymonori/TF-ESPCN

- x2, x3, x4 trained models available

- Advantage: It is tiny and fast, and still performs well.

- Disadvantage: Perform worse visually than newer, more robust models.

- Speed: < 0.01 sec for every scaling factor on 256x256 images on an Intel i7-9700K CPU.

- Original paper: [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](<https://arxiv.org/abs/1609.05158>) [2]

Trained models can be downloaded from [here](https://github.com/Saafke/FSRCNN_Tensorflow/tree/master/models).

- Size of the model: ~40KB (~9kb for FSRCNN-small)

- This model was trained for ~30 iterations with a batch size of 1

- Link to implementation code: https://github.com/Saafke/FSRCNN_Tensorflow

- Advantage: Fast, small and accurate

- Disadvantage: Not state-of-the-art accuracy

- Speed: < 0.01 sec for every scaling factor on 256x256 images on an Intel i7-9700K CPU.

- Notes: FSRCNN-small has fewer parameters, thus less accurate but faster.

- Original paper: [Accelerating the Super-Resolution Convolutional Neural Network](https://mmlab.ie.cuhk.edu.hk/projects/FSRCNN.html) [3]

Trained models can be downloaded from [here](https://github.com/fannymonori/TF-LapSRN/tree/master/export).

- Size of the model: between 1-5Mb

- This model was trained for ~50 iterations with a batch size of 32

- Link to implementation code: https://github.com/fannymonori/TF-LAPSRN

- x2, x4, x8 trained models available

- Advantage: The model can do multi-scale super-resolution with one forward pass. It can now support 2x, 4x, 8x, and [2x, 4x] and [2x, 4x, 8x] super-resolution.

- Disadvantage: It is slower than ESPCN and FSRCNN, and the accuracy is worse than EDSR.

- Speed: < 0.1 sec for every scaling factor on 256x256 images on an Intel i7-9700K CPU.

- Original paper: [Deep laplacian pyramid networks for fast and accurate super-resolution](<https://arxiv.org/abs/1710.01992>) [4]

Comparing different algorithms. Scale x4 on monarch.png.

| | Inference time in seconds (CPU)| PSNR | SSIM |

| ------------- |:-------------------:| ---------:|--------:|

| ESPCN |0.01159 | 26.5471 | 0.88116 |

| EDSR |3.26758 |**29.2404** |**0.92112** |

| FSRCNN | 0.01298 | 26.5646 | 0.88064 |

| LapSRN |0.28257 |26.7330 |0.88622 |

| Bicubic |0.00031 |26.0635 |0.87537 |

| Nearest neighbor |**0.00014** |23.5628 |0.81741 |

| Lanczos |0.00101 |25.9115 |0.87057 |

[1] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee, **"Enhanced Deep Residual Networks for Single Image Super-Resolution"**, <i> 2nd NTIRE: New Trends in Image Restoration and Enhancement workshop and challenge on image super-resolution in conjunction with **CVPR 2017**. </i> [[PDF](https://openaccess.thecvf.com/content_cvpr_2017_workshops/w12/papers/Lim_Enhanced_Deep_Residual_CVPR_2017_paper.pdf)] [[arXiv](https://arxiv.org/abs/1707.02921)] [[Slide](https://cv.snu.ac.kr/research/EDSR/Presentation_v3(release).pptx)]

[2] Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A., Bishop, R., Rueckert, D. and Wang, Z., **"Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network"**, <i>Proceedings of the IEEE conference on computer vision and pattern recognition</i> **CVPR 2016**. [[PDF](https://openaccess.thecvf.com/content_cvpr_2016/papers/Shi_Real-Time_Single_Image_CVPR_2016_paper.pdf)] [[arXiv](https://arxiv.org/abs/1609.05158)]

[3] Chao Dong, Chen Change Loy, Xiaoou Tang. **"Accelerating the Super-Resolution Convolutional Neural Network"**, <i> in Proceedings of European Conference on Computer Vision </i>**ECCV 2016**. [[PDF](https://personal.ie.cuhk.edu.hk/~ccloy/files/eccv_2016_accelerating.pdf)]

[[arXiv](https://arxiv.org/abs/1608.00367)] [[Project Page](https://mmlab.ie.cuhk.edu.hk/projects/FSRCNN.html)]

[4] Lai, W. S., Huang, J. B., Ahuja, N., and Yang, M. H., **"Deep laplacian pyramid networks for fast and accurate super-resolution"**, <i> In Proceedings of the IEEE conference on computer vision and pattern recognition </i>**CVPR 2017**. [[PDF](https://openaccess.thecvf.com/content_cvpr_2017/papers/Lai_Deep_Laplacian_Pyramid_CVPR_2017_paper.pdf)] [[arXiv](https://arxiv.org/abs/1710.01992)] [[Project Page](https://vllab.ucmerced.edu/wlai24/LapSRN/)]

#ifndef _OPENCV_DNN_SUPERRES_HPP_

#define _OPENCV_DNN_SUPERRES_HPP_

#include "opencv2/core.hpp"

#include "opencv2/dnn.hpp"

namespace cv

{

namespace dnn_superres

{

class CV_EXPORTS DnnSuperResImpl

{

/** @brief Net which holds the desired neural network

dnn::Net net;

std::string alg; //algorithm

int sc; //scale factor

void preprocess(InputArray inpImg, OutputArray outpImg);

void reconstruct_YCrCb(InputArray inpImg, InputArray origImg, OutputArray outpImg, int scale);

void reconstruct_YCrCb(InputArray inpImg, InputArray origImg, OutputArray outpImg);

void preprocess_YCrCb(InputArray inpImg, OutputArray outpImg);

/** @brief Empty constructor

DnnSuperResImpl();

/** @brief Constructor which immediately sets the desired model

DnnSuperResImpl(const std::string& algo, int scale);

/** @brief Read the model from the given path

void readModel(const std::string& path);

/** @brief Read the model from the given path

void readModel(const std::string& weights, const std::string& definition);

/** @brief Set desired model

void setModel(const std::string& algo, int scale);

/** @brief Upsample via neural network

void upsample(InputArray img, OutputArray result);

/** @brief Upsample via neural network of multiple outputs

void upsampleMultioutput(InputArray img, std::vector<Mat> &imgs_new, const std::vector<int>& scale_factors, const std::vector<String>& node_names);

/** @brief Returns the scale factor of the model:

int getScale();

/** @brief Returns the scale factor of the model:

std::string getAlgorithm();

};

}} // cv::dnn_superres::

#endif

#include "perf_precomp.hpp"

using namespace std;

using namespace cv;

using namespace perf;

namespace opencv_test { namespace {

typedef perf::TestBaseWithParam<tuple<tuple<string,string,int>, string> > dnn_superres;

#define MODEL testing::Values(tuple<string,string,int> {"espcn","ESPCN_x2.pb",2}, \

tuple<string,string,int> {"lapsrn","LapSRN_x4.pb",4})

#define IMAGES testing::Values("cv/dnn_superres/butterfly.png", "cv/shared/baboon.png", "cv/shared/lena.png")

const string TEST_DIR = "cv/dnn_superres";

PERF_TEST_P(dnn_superres, upsample, testing::Combine(MODEL, IMAGES))

{

tuple<string,string,int> model = get<0>( GetParam() );

string image_name = get<1>( GetParam() );

string model_name = get<0>(model);

string model_filename = get<1>(model);

int scale = get<2>(model);

string model_path = getDataPath( TEST_DIR + "/" + model_filename );

string image_path = getDataPath( image_name );

DnnSuperResImpl sr;

sr.readModel(model_path);

sr.setModel(model_name, scale);

Mat img = imread(image_path);

Mat img_new(img.rows * scale, img.cols * scale, CV_8UC3);

declare.in(img, WARMUP_RNG).out(img_new).iterations(10);

TEST_CYCLE() { sr.upsample(img, img_new); }

SANITY_CHECK_NOTHING();

}

}}

#include "perf_precomp.hpp"

CV_PERF_TEST_MAIN( dnn_superres )

#ifndef __OPENCV_PERF_PRECOMP_HPP__

#define __OPENCV_PERF_PRECOMP_HPP__

#include "opencv2/ts.hpp"

#include "opencv2/dnn_superres.hpp"

namespace opencv_test {

using namespace cv::dnn_superres;

}

#endif