set(the_description "Image Quality Analysis API")

ocv_define_module(quality opencv_core opencv_imgproc WRAP python)

Quality API, Image Quality Analysis

Implementation of various image quality analysis (IQA) algorithms

- **Mean squared error (MSE)**

https://en.wikipedia.org/wiki/Mean_squared_error

- **Peak signal-to-noise ratio (PSNR)**

https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio

- **Structural similarity (SSIM)**

https://en.wikipedia.org/wiki/Structural_similarity

- **Gradient Magnitude Similarity Deviation (GMSD)**

https://www4.comp.polyu.edu.hk/~cslzhang/IQA/GMSD/GMSD.htm

In general, the GMSD algorithm should yield the best result for full-reference IQA.

Interface/Usage

All algorithms can be accessed through the simpler static `compute` methods,

or be accessed by instance created via the static `create` methods.

Instance methods are designed to be more performant when comparing one source

file against multiple comparison files, as the algorithm-specific preprocessing on the

source file need not be repeated with each call.

For performance reaasons, it is recommended, but not required, for users of this module

to convert input images to grayscale images prior to processing.

SSIM and GMSD were originally tested by their respective researchers on grayscale uint8 images,

but this implementation will compute the values for each channel if the user desires to do so.

Quick Start/Usage

#include <opencv2/quality.hpp>

cv::Mat img1, img2; /* your cv::Mat images */

std::vector<cv::Mat> quality_maps; /* output quality map(s) (optional) */

/* compute MSE via static method */

cv::Scalar result_static = quality::QualityMSE::compute(img1, img2, quality_maps); /* or cv::noArray() if not interested in output quality maps */

/* alternatively, compute MSE via instance */

cv::Ptr<quality::QualityBase> ptr = quality::QualityMSE::create(img1);

cv::Scalar result = ptr->compute( img2 ); /* compute MSE, compare img1 vs img2 */

ptr->getQualityMaps(quality_maps); /* optionally, access output quality maps */

Library Design

Each implemented algorithm shall:

- Inherit from `QualityBase`, and properly implement/override `compute`, `empty` and `clear` instance methods, along with a static `compute` method.

- Accept one or more `cv::Mat` or `cv::UMat` via `InputArrayOfArrays` for computation. Each input `cv::Mat` or `cv::UMat` may contain one or more channels. If the algorithm does not support multiple channels or multiple inputs, it should be documented and an appropriate assertion should be in place.

- Return a `cv::Scalar` with per-channel computed value. If multiple input images are provided, the resulting scalar should return the average result per channel.

- Compute result via a single, static method named `compute` and via an overridden instance method (see `compute` in `qualitybase.hpp`).

- Perform any setup and/or pre-processing of reference images in the constructor, allowing for efficient computation when comparing the reference image(s) versus multiple comparison image(s). No-reference algorithms should accept images for evaluation in the `compute` method.

- Optionally compute resulting quality maps. Instance `compute` method should store them in `QualityBase::_qualityMaps` as the mat type defined by `QualityBase::_quality_map_type`, or override `QualityBase::getQualityMaps`. Static `compute` method should return them in an `OutputArrayOfArrays` parameter.

- Document algorithm in this readme and in its respective header. Documentation should include interpretation for the results of `compute` as well as the format of the output quality maps (if supported), along with any other notable usage information.

- Implement tests of static `compute` method and instance methods using single- and multi-channel images, multi-frame images, and OpenCL enabled and disabled

To Do

- Document the output quality maps for each algorithm

- Implement at least one no-reference IQA algorithm

- Investigate precision loss with cv::Filter2D + UMat + CV_32F + OCL for GMSD

#ifndef OPENCV_QUALITY_HPP

#define OPENCV_QUALITY_HPP

#include "quality/qualitybase.hpp"

#include "quality/qualitymse.hpp"

#include "quality/qualitypsnr.hpp"

#include "quality/qualityssim.hpp"

#include "quality/qualitygmsd.hpp"

#endif

#ifndef OPENCV_QUALITY_QUALITY_UTILS_HPP

#define OPENCV_QUALITY_QUALITY_UTILS_HPP

#include <limits> // numeric_limits

#include "qualitybase.hpp"

namespace cv

{

namespace quality

{

namespace quality_utils

{

static CV_CONSTEXPR const int EXPANDED_MAT_DEFAULT_TYPE = CV_32F;

template <typename OutT, typename InT>

inline OutT expand_mat(const InT& src, int TYPE_DEFAULT = EXPANDED_MAT_DEFAULT_TYPE)

{

OutT result = {};

// by default, expand to 32F unless we already have >= 32 bits, then go to 64

// if/when we can detect OpenCL CV_16F support, opt for that when input depth == 8

// note that this may impact the precision of the algorithms and would need testing

int type = TYPE_DEFAULT;

switch (src.depth())

{

case CV_32F:

case CV_32S:

case CV_64F:

type = CV_64F;

}; // switch

src.convertTo(result, type);

return result;

}

template <typename R>

inline std::vector<R> expand_mats(InputArrayOfArrays arr, int TYPE_DEFAULT = EXPANDED_MAT_DEFAULT_TYPE)

{

std::vector<R> result = {};

std::vector<UMat> umats = {};

std::vector<Mat> mats = {};

if (arr.isUMatVector())

arr.getUMatVector(umats);

else if (arr.isUMat())

umats.emplace_back(arr.getUMat());

else if (arr.isMatVector())

arr.getMatVector(mats);

else if (arr.isMat())

mats.emplace_back(arr.getMat());

else

CV_Error(Error::StsNotImplemented, "Unsupported input type");

// convert umats, mats to expanded internal type

for (auto& umat : umats)

result.emplace_back(expand_mat<R>(umat, TYPE_DEFAULT ));

for (auto& mat : mats)

result.emplace_back(expand_mat<R>(mat, TYPE_DEFAULT ));

return result;

}

inline double mse_to_psnr(double mse, double max_pixel_value)

{

return (mse == 0.)

? std::numeric_limits<double>::infinity()

: 10. * std::log10((max_pixel_value * max_pixel_value) / mse)

;

}

inline cv::Scalar mse_to_psnr(cv::Scalar mse, double max_pixel_value)

{

for (int i = 0; i < mse.rows; ++i)

mse(i) = mse_to_psnr(mse(i), max_pixel_value);

return mse;

}

} // quality_utils

} // quality

} // cv

#endif

#ifndef OPENCV_QUALITYBASE_HPP

#define OPENCV_QUALITYBASE_HPP

#include <vector>

#include <opencv2/core.hpp>

namespace cv

{

namespace quality

{

class CV_EXPORTS_W QualityBase

: public virtual Algorithm

{

/** @brief Destructor */

virtual ~QualityBase() = default;

/**

virtual CV_WRAP cv::Scalar compute( InputArrayOfArrays cmpImgs ) = 0;

/** @brief Returns output quality map images that were generated during computation, if supported by the algorithm */

virtual CV_WRAP void getQualityMaps(OutputArrayOfArrays dst) const

{

if (!dst.needed() || _qualityMaps.empty() )

return;

auto qMaps = InputArray(_qualityMaps);

dst.create(qMaps.size(), qMaps.type());

dst.assign(_qualityMaps);

}

/** @brief Implements Algorithm::clear() */

CV_WRAP void clear() CV_OVERRIDE { _qualityMaps.clear(); Algorithm::clear(); }

/** @brief Implements Algorithm::empty() */

CV_WRAP bool empty() const CV_OVERRIDE { return _qualityMaps.empty(); }

/** @brief internal quality map type default */

using _quality_map_type = cv::UMat;

/** @brief Output quality maps if generated by algorithm */

std::vector<_quality_map_type> _qualityMaps;

}; // QualityBase

} // quality

} // cv

#endif

#ifndef OPENCV_QUALITY_QUALITYGMSD_HPP

#define OPENCV_QUALITY_QUALITYGMSD_HPP

#include "qualitybase.hpp"

namespace cv

{

namespace quality

{

class CV_EXPORTS_W QualityGMSD

: public QualityBase {

/**

CV_WRAP cv::Scalar compute(InputArrayOfArrays cmpImgs) CV_OVERRIDE;

/** @brief Implements Algorithm::empty() */

CV_WRAP bool empty() const CV_OVERRIDE { return _refImgData.empty() && QualityBase::empty(); }

/** @brief Implements Algorithm::clear() */

CV_WRAP void clear() CV_OVERRIDE { _refImgData.clear(); QualityBase::clear(); }

/**

CV_WRAP static Ptr<QualityGMSD> create(InputArrayOfArrays refImgs);

/**

CV_WRAP static cv::Scalar compute(InputArrayOfArrays refImgs, InputArrayOfArrays cmpImgs, OutputArrayOfArrays qualityMaps);

// holds computed values for an input mat

struct _mat_data

{

using mat_type = QualityBase::_quality_map_type;

mat_type

gradient_map

, gradient_map_squared

;

_mat_data(const mat_type&);

// converts mat/umat to vector of mat_data

static std::vector<_mat_data> create(InputArrayOfArrays arr);

// compute for a single frame

static std::pair<cv::Scalar, mat_type> compute(const _mat_data& lhs, const _mat_data& rhs);

// compute for vector of inputs

static cv::Scalar compute(const std::vector<_mat_data>& lhs, const std::vector<_mat_data>& rhs, OutputArrayOfArrays qualityMaps);

}; // mat_data

/** @brief Reference image data */

std::vector<_mat_data> _refImgData;

/**

QualityGMSD(std::vector<_mat_data> refImgData)

: _refImgData(std::move(refImgData))

{}

}; // QualityGMSD

} // quality

} // cv

#endif