DNN: Add New API blobFromImageParam #22750

zihaomu · 2022-11-04T02:56:26Z

The purpose of this PR:

Add new API blobFromImageParam to extend blobFromImage API. It can support the different datalayout (NCHW or NHWC), and letter_box.
~~blobFromImage can output CV_16F~~

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

zihaomu · 2022-11-24T08:34:10Z

Hi, the updated code has re-used old blobFromImage API, only change is convert scalefactor from double to Scalar. And this change brokes the java binding, shown as follows:

    [mkdir] Created dir: /home/ci/build/java_test/build/classes
    [javac] Compiling 58 source files to /home/ci/build/java_test/build/classes
    [javac] /home/ci/build/java_test/src/org/opencv/test/dnn/DnnListRegressionTest.java:69: error: incompatible types: double cannot be converted to Scalar
    [javac]         Mat inputBlob = Dnn.blobFromImage(image, 1.0, new Size(224, 224), new Scalar(0), true, true);
    [javac]                                                  ^
    [javac] /home/ci/build/java_test/src/org/opencv/test/dnn/DnnTensorFlowTest.java:98: error: incompatible types: double cannot be converted to Scalar
    [javac]         Mat inputBlob = Dnn.blobFromImage(image, 1.0, new Size(224, 224), new Scalar(0), true, true);
    [javac]                                                  ^
    [javac] Note: /home/ci/build/java_test/src/org/opencv/test/dnn/DnnTensorFlowTest.java uses or overrides a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] Note: Some messages have been simplified; recompile with -Xdiags:verbose to get full output
    [javac] 2 errors

Is there any solution?
Is it possible to map one cpp API of blobFromImage into two java APi?

modules/dnn/include/opencv2/dnn/dnn.hpp

alalek

As we have a new version with "parameters" do we really need to update old versions blobFromImage()? We could deprecate them and suggest using a new variant.

modules/dnn/include/opencv2/dnn/dnn.hpp

asmorkalov · 2022-11-25T13:17:51Z

You can generate extra overloads in Java with ManualFuncs option of Java bindigs generator. See modules/core/misc/java/gen_dict.json as example. Java tests for all manual functions are required.

zihaomu · 2023-02-15T10:33:38Z

As we have a new version with "parameters" do we really need to update old versions blobFromImage()? We could deprecate them and suggest using a new variant.

Agree with @alalek. I propose that this PR only add new features to the new API blobFromImageParam, without changing the scalefactor type at old API (blobFromImage). Since the blobFromImage has too many arguments, and the future new features will be added to the blobFromImageParam.

modules/dnn/include/opencv2/dnn/dnn.hpp

modules/dnn/src/tensorflow/tf_importer.cpp

asmorkalov · 2023-03-22T06:40:23Z

@zihaomu @vpisarev friendly reminder.

modules/dnn/src/tflite/tflite_importer.cpp

modules/dnn/include/opencv2/dnn/dnn.hpp

modules/dnn/test/test_misc.cpp

asmorkalov

👍

vpisarev · 2023-04-19T06:27:04Z

@dkurt, can you please tell if "planar" layout used in TFLite importer is the same thing as "ND" layout used in other formats?

dkurt · 2023-04-19T07:41:30Z

Do we really need to introduce legacy TensorFlow NHWC data layout? I think it may introduce more confusion than strict NCHW input.

Also, I'm not sure about preserving ratio during scale (crop/letter_box). They are different in different applications. For example, EfficientDet from TensorFlow:

h, w = img.shape[0:2]
# 1. Resize and keep aspect ratio
assert(w >= h)
h = int(h / w * args.height)
inp = cv.resize(img.astype(np.float32), (args.width, h))  # It's important to perform resize for fp32
# 2. Zero padding to the bottom
inp = np.pad(inp, ((0, args.height - h), (0, 0), (0, 0)), 'constant')
inp = np.expand_dims(inp.transpose(2, 0, 1), axis=0)

source

@dkurt, can you please tell if "planar" layout used in TFLite importer is the same thing as "ND" layout used in other formats?

@vpisarev, TFLite's planar is the same as TensorFlow's planar. It's used in case of 4D->3D or 2D reshaping when OpenCV data layout matches TensorFlow during import.

modules/dnn/include/opencv2/dnn/dnn.hpp

modules/dnn/src/dnn_utils.cpp

modules/dnn/include/opencv2/dnn/dnn.hpp

asmorkalov · 2023-04-21T10:25:47Z

[  FAILED  ] blobFromImageWithParams_4ch.letter_box

zihaomu · 2023-04-21T14:58:20Z

Hi @asmorkalov, thanks for your reminder, updated.

DNN: Add New API blobFromImageParam opencv#22750 The purpose of this PR: 1. Add new API `blobFromImageParam` to extend `blobFromImage` API. It can support the different data layout (NCHW or NHWC), and letter_box. 2. ~~`blobFromImage` can output `CV_16F`~~ ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [ ] There is a reference to the original bug report and related work - [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [ ] The feature is well documented and sample code can be built with the project CMake

zihaomu force-pushed the improve_blobFromImage branch 2 times, most recently from 7b47836 to e4abcec Compare November 4, 2022 05:53

zihaomu force-pushed the improve_blobFromImage branch 5 times, most recently from 60209c1 to d280eab Compare November 24, 2022 08:25

zihaomu requested a review from vpisarev November 25, 2022 06:03

zihaomu force-pushed the improve_blobFromImage branch from a5c6ceb to 520b472 Compare November 25, 2022 06:55

zihaomu commented Nov 25, 2022

View reviewed changes

modules/dnn/include/opencv2/dnn/dnn.hpp Outdated Show resolved Hide resolved

alalek reviewed Nov 25, 2022

View reviewed changes

modules/dnn/include/opencv2/dnn/dnn.hpp Outdated Show resolved Hide resolved

alalek reviewed Nov 25, 2022

View reviewed changes

modules/dnn/include/opencv2/dnn/dnn.hpp Show resolved Hide resolved

zihaomu force-pushed the improve_blobFromImage branch 4 times, most recently from b80264b to 716ba23 Compare November 30, 2022 03:54

zihaomu force-pushed the improve_blobFromImage branch 2 times, most recently from 972d8ce to 12acf76 Compare February 15, 2023 08:29

zihaomu changed the title ~~DNN: improve blobFromImage API~~ DNN: Extending blobFromImage through New API blobFromImageParam Feb 15, 2023

zihaomu changed the title ~~DNN: Extending blobFromImage through New API blobFromImageParam~~ DNN: Add New API blobFromImageParam Feb 15, 2023

zihaomu force-pushed the improve_blobFromImage branch from 12acf76 to a58ce0c Compare February 15, 2023 11:12

zihaomu commented Feb 17, 2023

View reviewed changes

modules/dnn/include/opencv2/dnn/dnn.hpp Outdated Show resolved Hide resolved

vpisarev reviewed Feb 17, 2023

View reviewed changes

modules/dnn/include/opencv2/dnn/dnn.hpp Outdated Show resolved Hide resolved

vpisarev reviewed Feb 17, 2023

View reviewed changes

modules/dnn/include/opencv2/dnn/dnn.hpp Outdated Show resolved Hide resolved

vpisarev reviewed Feb 17, 2023

View reviewed changes

modules/dnn/include/opencv2/dnn/dnn.hpp Outdated Show resolved Hide resolved

vpisarev reviewed Feb 17, 2023

View reviewed changes

modules/dnn/include/opencv2/dnn/dnn.hpp Outdated Show resolved Hide resolved

zihaomu force-pushed the improve_blobFromImage branch from edd832e to b8d1f84 Compare March 1, 2023 03:48

zihaomu commented Mar 1, 2023

View reviewed changes

modules/dnn/src/tensorflow/tf_importer.cpp Outdated Show resolved Hide resolved

zihaomu force-pushed the improve_blobFromImage branch 3 times, most recently from 6816dbd to 5c0ed4f Compare April 12, 2023 07:22

vpisarev reviewed Apr 14, 2023

View reviewed changes

modules/dnn/src/tflite/tflite_importer.cpp Outdated Show resolved Hide resolved

zihaomu force-pushed the improve_blobFromImage branch 2 times, most recently from b386d90 to 957cd82 Compare April 17, 2023 03:35

asmorkalov reviewed Apr 17, 2023

View reviewed changes

extend blobFromImage API by blobFromImageWithParams

b4f6383

zihaomu force-pushed the improve_blobFromImage branch from 957cd82 to b4f6383 Compare April 18, 2023 05:44

asmorkalov approved these changes Apr 18, 2023

View reviewed changes

vpisarev self-requested a review April 19, 2023 06:23

vpisarev approved these changes Apr 19, 2023

View reviewed changes

asmorkalov reviewed Apr 20, 2023

View reviewed changes

modules/dnn/include/opencv2/dnn/dnn.hpp Outdated Show resolved Hide resolved

modules/dnn/include/opencv2/dnn/dnn.hpp Outdated Show resolved Hide resolved

modules/dnn/src/dnn_utils.cpp Outdated Show resolved Hide resolved

zihaomu commented Apr 21, 2023

View reviewed changes

modules/dnn/include/opencv2/dnn/dnn.hpp Show resolved Hide resolved

zihaomu force-pushed the improve_blobFromImage branch from a8c3f22 to 30aeda5 Compare April 21, 2023 09:25

add more details for different layout and remove the hack code.

9450d87

zihaomu force-pushed the improve_blobFromImage branch from 30aeda5 to 9450d87 Compare April 21, 2023 14:57

asmorkalov assigned vpisarev Apr 21, 2023

asmorkalov merged commit 601778e into opencv:4.x Apr 21, 2023

asmorkalov added this to the 4.8.0 milestone Apr 21, 2023

asmorkalov mentioned this pull request May 31, 2023

(5.x) Merge 4.x #23718

Merged

asmorkalov mentioned this pull request Oct 30, 2023

blobFromImage and crop #14659

Closed

Uh oh!

DNN: Add New API blobFromImageParam #22750

DNN: Add New API blobFromImageParam #22750

Uh oh!

Conversation

zihaomu commented Nov 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

zihaomu commented Nov 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

asmorkalov commented Nov 25, 2022

Uh oh!

zihaomu commented Feb 15, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asmorkalov commented Mar 22, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asmorkalov left a comment

Choose a reason for hiding this comment

Uh oh!

vpisarev commented Apr 19, 2023

Uh oh!

dkurt commented Apr 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asmorkalov commented Apr 21, 2023

Uh oh!

zihaomu commented Apr 21, 2023

Uh oh!

Uh oh!

zihaomu commented Nov 4, 2022 •

edited

Loading

zihaomu commented Nov 24, 2022 •

edited

Loading

dkurt commented Apr 19, 2023 •

edited

Loading