Switch to new OpenVINO API after 2022.1 release #22957

dkurt · 2022-12-14T08:24:53Z

Use OpenVINO Runtime API (aka Tensor API or API 2.0) which has been introduces in 2022.1
Remove InferenceEngine::DataPtr completely
Internal dynamism feature works only with a new API. Actual for Faster-* family models and with layers such as NonZero, Top-k and others. This feature may improve performance.

Tested with OpenVINO 2022.1 (CI)
Tested with OpenVINO 2021.4 (CI)
Tested with old OpenVINO (for example, 2020.3) - (alalek: older versions are not supported on 4.x branch)

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

force_builders=Custom
Xbuild_image:Custom=ubuntu-openvino-2021.4.2:20.04
build_image:Custom=ubuntu-openvino-2022.1.0:20.04
test_modules:Custom=dnn,python2,python3,java,gapi,video
buildworker:Custom=linux-1
# disabled due high memory usage: test_opencl:Custom=ON
test_opencl:Custom=ON
test_bigdata:Custom=1
test_filter:Custom=*
allow_multiple_commits=1

modules/dnn/src/net_openvino.cpp

modules/dnn/src/ie_ngraph.cpp

alalek · 2022-12-20T00:45:12Z

@dkurt Thank you for the update!

CI failures are related to OOM issue (32Gb).

./bin/opencv_test_dnn --gtest_filter=*arknet*

consumes 17Gb with OpenVINO 2022.1 and enabled OpenCL.

Could you have a chance to take a look on this?

dkurt · 2022-12-22T11:35:37Z

There must be a leak. Can reproduce with a single test on both CPU and GPU:

/usr/bin/time --verbose ./opencv_test_dnn --gtest_repeat=1 --gtest_param_filter=NGRAPH/OCL --gtest_filter=Test_Darknet_nets.YOLOv4x_mish/*

Num repeats	Top resident memory (NGRAPH/OCL)	Top resident memory (NGRAPH/CPU)
1	2.14GB	3.59GB
2	3.21GB	6.15GB
3	4.27GB	8.71GB

Update: found two problems: asynchronous callback and cycled dependency between InfEngineNgraphNode and InfEngineNgraphNet caused by requestedOutputs so InfEngineNgraphNet destructor was not called.

alalek

@dkurt Thank you 👍

dkurt · 2022-12-22T19:36:32Z

Need to fix failed OCL tests with 2021.4, thanks for configuring.

@alalek, problem is reproduced on 4.x branch. Appears on HETERO:GPU,CPU configuration and fixed in 2022.x OpenVINO release. Solved by adding missed layers so no fallback to CPU is needed.

…Net. Add InferRequest callback only for async inference. Do not capture InferRequest object.

JulienMaille · 2022-12-24T00:28:30Z

@alalek is it too soon to try compiling with newly released Openvino?

Switch to new OpenVINO API after 2022.1 release * Pass Layer_Test_Convolution_DLDT.Accuracy/0 test * Pass test Test_Caffe_layers.Softmax * Failed 136 tests * Fix Concat. Failed 120 tests * Custom nGraph ops. 19 failed tests * Set and get properties from Core * Read model from buffer * Change MaxPooling layer output names. Restore reshape * Cosmetic changes * Cosmetic changes * Override getOutputsInfo * Fixes for OpenVINO < 2022.1 * Async inference for 2021.4 and less * Compile model with config * Fix serialize for 2022.1 * Asynchronous inference with 2022.1 * Handle 1d outputs * Work with model with dynamic output shape * Fixes with 1d output for old API * Control outputs by nGraph function for all OpenVINO versions * Refer inputs in PrePostProcessor by indices * Fix cycled dependency between InfEngineNgraphNode and InfEngineNgraphNet. Add InferRequest callback only for async inference. Do not capture InferRequest object. * Fix tests thresholds * Fix HETERO:GPU,CPU plugin issues with unsupported layer

dkurt commented Dec 14, 2022

View reviewed changes

modules/dnn/src/net_openvino.cpp Outdated Show resolved Hide resolved

dkurt force-pushed the new_openvino_api branch from 7b02fee to 0c253d0 Compare December 15, 2022 14:28

dkurt marked this pull request as ready for review December 16, 2022 14:19

dkurt commented Dec 19, 2022

View reviewed changes

modules/dnn/src/ie_ngraph.cpp Show resolved Hide resolved

dkurt force-pushed the new_openvino_api branch from 47bd8d5 to 3296679 Compare December 22, 2022 15:01

alalek approved these changes Dec 22, 2022

View reviewed changes

dkurt added 21 commits December 23, 2022 11:32

Pass Layer_Test_Convolution_DLDT.Accuracy/0 test

0ba593c

Pass test Test_Caffe_layers.Softmax

ebea45d

Failed 136 tests

ab940cd

Fix Concat. Failed 120 tests

b252ab6

Custom nGraph ops. 19 failed tests

cbf1458

Set and get properties from Core

10f58c3

Read model from buffer

1f30310

Change MaxPooling layer output names. Restore reshape

83149ee

Cosmetic changes

37f7d68

Cosmetic changes

855e759

Override getOutputsInfo

deb8a3c

Fixes for OpenVINO < 2022.1

3060c06

Async inference for 2021.4 and less

4f47b92

Compile model with config

e94c53d

Fix serialize for 2022.1

f53b6c5

Asynchronous inference with 2022.1

ecfc066

Handle 1d outputs

f4a8a76

Work with model with dynamic output shape

e26d083

Fixes with 1d output for old API

020678f

Control outputs by nGraph function for all OpenVINO versions

2a50a7b

Refer inputs in PrePostProcessor by indices

21880ee

dkurt added 2 commits December 23, 2022 11:32

Fix cycled dependency between InfEngineNgraphNode and InfEngineNgraph…

442fd91

…Net. Add InferRequest callback only for async inference. Do not capture InferRequest object.

Fix tests thresholds

bd2ab10

dkurt force-pushed the new_openvino_api branch from b209c8d to 4f3e8d8 Compare December 23, 2022 08:34

Fix HETERO:GPU,CPU plugin issues with unsupported layer

1c56361

dkurt force-pushed the new_openvino_api branch from 4f3e8d8 to 1c56361 Compare December 23, 2022 08:53

alalek merged commit 8681686 into opencv:4.x Dec 23, 2022

alalek mentioned this pull request Dec 28, 2022

OpenCV cannot be compiled in debug mode with OpenVINO #23048

Closed

4 tasks

alalek mentioned this pull request Jan 8, 2023

(5.x) Merge 4.x #23113

Merged

dkurt mentioned this pull request Mar 30, 2023

Error trying to build OpenCV 4.5.5 with OpenVINO 2023.2 #23054

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Switch to new OpenVINO API after 2022.1 release #22957

Switch to new OpenVINO API after 2022.1 release #22957

Uh oh!

dkurt commented Dec 14, 2022 •

edited by alalek

Loading

Uh oh!

Uh oh!

Uh oh!

alalek commented Dec 20, 2022

Uh oh!

dkurt commented Dec 22, 2022 •

edited

Loading

Uh oh!

alalek left a comment

Uh oh!

dkurt commented Dec 22, 2022 •

edited

Loading

Uh oh!

JulienMaille commented Dec 24, 2022

Uh oh!

Uh oh!

Uh oh!

Switch to new OpenVINO API after 2022.1 release #22957

Switch to new OpenVINO API after 2022.1 release #22957

Uh oh!

Conversation

dkurt commented Dec 14, 2022 • edited by alalek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

Uh oh!

Uh oh!

alalek commented Dec 20, 2022

Uh oh!

dkurt commented Dec 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

dkurt commented Dec 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JulienMaille commented Dec 24, 2022

Uh oh!

Uh oh!

dkurt commented Dec 14, 2022 •

edited by alalek

Loading

dkurt commented Dec 22, 2022 •

edited

Loading

dkurt commented Dec 22, 2022 •

edited

Loading