Add DNN-based face detection and face recognition into modules/objdetect #20422

fengyuentau · 2021-07-16T08:20:30Z

Merge with extra: opencv/opencv_extra#903

This PR adds a DNN-based face detector and a DNN-based face recognizer into modules/objdetect.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

samples/cpp/face_match.cpp

berak · 2021-07-17T09:21:17Z

modules/objdetect/src/face_recognize.cpp

+    {
+        Mat inputBolb = dnn::blobFromImage(_aligned_img, 1, Size(112, 112), Scalar(0, 0, 0), true, false);
+        this->model.setInput(inputBolb);
+        return this->model.forward(_face_feature);


imho it needs a clone() here, else the 2nd run of the network overwrites the 1st
(shallow copy of the network's last layer)
and both features are the same

heh, you had it correct in your previous version

Sorry for the late response. I deleted the '.clone()' because I think maintaining the extracted feature is up to the user. User may allow network to overwrite the last feature to reduce excess memory usage.
And I'm sorry for another mistake, this function need not return any value.

Maybe, I can delete 'return', and add '.clone()' into the sample code.

faceRecognizer->facefeature(aligned_face1, feature1); feature1 = feature1.clone(); faceRecognizer->facefeature(aligned_face2, feature2); feature2 = feature2.clone();

samples/cpp/face_match.cpp

modules/objdetect/include/opencv2/objdetect/face.hpp

modules/objdetect/src/face_recognize.cpp

modules/objdetect/include/opencv2/objdetect/face.hpp

modules/objdetect/src/face_detect.cpp

vpisarev · 2021-07-19T06:43:45Z

@fengyuentau, thank you! Please, also add some regression test

modules/objdetect/src/face_detect.cpp

vpisarev · 2021-09-07T06:25:49Z

@fengyuentau, looks good to me, thank you!
@alalek, do you have any other requests regarding this PR? or we can merge it?

alalek

@vpisarev What is about reusing of High Level DNN API concept? How to manage execution of used networks - select backend/device at least?

How this code should be aligned with existed opencv_face_detector?

samples/dnn/face_match.py

alalek · 2021-09-07T07:34:40Z

samples/dnn/face_detect.py

+    detector = cv.FaceDetectorYN.create(
+        args.model,
+        "",
+        (320, 320),


(320, 320)

How is this aligned with .setInputSize()?

This algorithm requires to generate priors/anchors for different input sizes. My intention is to allow users to set the input size for the network when creating instances in case the input size is fixed, such as taking video stream as input. However, users may also want to detect on images of different sizes with one instance, so .setInputSize() is added to do so without distorting images.

alalek · 2021-09-07T07:36:22Z

samples/dnn/face_detect.py

+        image = cv.imread(args.input)
+
+        # Set input size before inference
+        detector.setInputSize((image.shape[1], image.shape[0]))


Input of what? frame? network input to be preprocessed?

Class methods have zero documentation.

I added the documentation to this class method in the latest commit. This class method sets the size of network input, which overwrites the input size in create.

modules/objdetect/src/face_recognize.cpp

* fix typos in samples/dnn/face_match.py * import numpy before importing cv2 * add documentation to .setInputSize() * remove extra include in face_recognize.cpp

fengyuentau · 2021-09-10T03:11:53Z

@vpisarev What is about reusing of High Level DNN API concept? How to manage execution of used networks - select backend/device at least?

How this code should be aligned with existed opencv_face_detector?

I read some code and documents on the TextDetectionModel and TextRegconitionModel APIs. Does your "High Level DNN API" mean something like those two APIs? If so, I think we can make two base classes named FaceDetectionModel and FaceRecognitionModel respectively, and have existing APIs inherited from them.

vpisarev · 2021-09-13T09:18:39Z

@fengyuentau, we discussed this PR another time. We reached consensus that the face detector class cannot reuse the object detection class from dnn. But could you, please, derive the face detection and recognition classes from cv::dnn::Model?

alalek · 2021-09-13T11:24:09Z

from cv::dnn::Model

BTW, It would be easy in case of code in dnn module. But currently this code is in objdetect module. Internal dnn::Model::Impl interface is no exposing from dnn to other modules for now - more refactoring is required (probably from OpenCV Core Dev Team).

Can we put these new algorithms into namespace inline experimental { } and improve them after release?

vpisarev · 2021-09-15T00:35:01Z

@alalek, I suggest to follow the same approach as with the deep learning-based tracking algorithms in opencv_video. Putting algorithms into a nested namespace is a bad idea, I think.

If it will take a lot of time to refactor dnn to expose dnn::Model properly, I suggest to merge this pull request as-is

fengyuentau · 2021-09-15T08:52:52Z

jenkins cn please retry a build

fengyuentau · 2021-09-22T09:24:49Z

jenkins cn please retry a build

fengyuentau · 2021-09-28T03:18:50Z

@alalek Anything else needs to be done for this PR regarding your and Vadim's comment? Can you help reviewing again? Thanks.

modules/objdetect/src/face_recognize.cpp

samples/dnn/face_match.cpp

samples/dnn/face_match.py

asenyaev · 2021-09-29T10:35:37Z

jenkins cn please retry a build

alalek · 2021-10-04T18:16:26Z

See comment in opencv_extra's PR.

Add DNN-based face detection and face recognition into modules/objdetect * Add DNN-based face detector impl and interface * Add a sample for DNN-based face detector * add recog * add notes * move samples from samples/cpp to samples/dnn * add documentation for dnn_face * add set/get methods for input size, nms & score threshold and topk * remove the DNN prefix from the face detector and face recognizer * remove default values in the constructor of impl * regenerate priors after setting input size * two filenames for readnet * Update face.hpp * Update face_recognize.cpp * Update face_match.cpp * Update face.hpp * Update face_recognize.cpp * Update face_match.cpp * Update face_recognize.cpp * Update dnn_face.markdown * Update dnn_face.markdown * Update face.hpp * Update dnn_face.markdown * add regression test for face detection * remove underscore prefix; fix warnings * add reference & acknowledgement for face detection * Update dnn_face.markdown * Update dnn_face.markdown * Update ts.hpp * Update test_face.cpp * Update face_match.cpp * fix a compile error for python interface; add python examples for face detection and recognition * Major changes for Vadim's comments: * Replace class name FaceDetector with FaceDetectorYN in related failes * Declare local mat before loop in modules/objdetect/src/face_detect.cpp * Make input image and save flag optional in samples/dnn/face_detect(.cpp, .py) * Add camera support in samples/dnn/face_detect(.cpp, .py) * correct file paths for regression test * fix convertion warnings; remove extra spaces * update face_recog * Update dnn_face.markdown * Fix warnings and errors for the default CI reports: * Remove trailing white spaces and extra new lines. * Fix convertion warnings for windows and iOS. * Add braces around initialization of subobjects. * Fix warnings and errors for the default CI systems: * Add prefix 'FR_' for each value name in enum DisType to solve the redefinition error for iOS compilation; Modify other code accordingly * Add bookmark '#tutorial_dnn_face' to solve warnings from doxygen * Correct documentations to solve warnings from doxygen * update FaceRecognizerSF * Fix the error for CI to find ONNX models correctly * add suffix f to float assignments * add backend & target options for initializing face recognizer * add checkeq for checking input size and preset size * update test and threshold * changes in response to alalek's comments: * fix typos in samples/dnn/face_match.py * import numpy before importing cv2 * add documentation to .setInputSize() * remove extra include in face_recognize.cpp * fix some bugs * Update dnn_face.markdown * update thresholds; remove useless code * add time suffix to YuNet filename in test * objdetect: update test code

fengyuentau and others added 4 commits July 15, 2021 14:36

Add DNN-based face detector impl and interface

fce310a

Add a sample for DNN-based face detector

3b2f34f

add recog

33ff54d

add notes

9152385

berak reviewed Jul 17, 2021

View reviewed changes

modules/objdetect/src/face_recognize.cpp Outdated Show resolved Hide resolved

move samples from samples/cpp to samples/dnn

e9ce233