G-API: Introduce ONNX backend for Inference #18716

dmatveev · 2020-11-03T08:06:02Z

Basic operations are implemented (Infer, -ROI, -List, -List2);
Implemented automatic preprocessing for ONNX models;
Test suite is extended with OPENCV_GAPI_ONNX_MODEL_PATH env for test data
(test data is an ONNX Model Zoo repo snapshot);
Fixed kernel lookup logic in core G-API:
- Lookup NN kernels not in the default package, but in the associated
  backend's aux package. Now two NN backends can work in the same graph.
Added Infer SSD demo and a combined ONNX/IE demo;

How to build:

Build & install the ONNX RT (currently tested with v1.5.1):

$ git clone --recursive https://github.com/microsoft/onnxruntime.git
$ cd onnxruntime
$ git checkout v1.5.1
$ git submodule update --init
$ ./build.sh --config Release --build_shared_lib --parallel \
$     --cmake_extra_defines CMAKE_INSTALL_PREFIX=install
$ cd build/Linux/Release
$ make install

...then specify extra options to OpenCV CMake:

$ cmake /path/to/opencv -DWITH_ONNX=ON -DORT_INSTALL_DIR=path-to-onnxrt-install

How to test:

Clone ONNX Model Zoo & selected models

git clone --recursive https://github.com/onnx/models.git
cd models
git lfs pull --include=vision/object_detection_segmentation/ssd-mobilenetv1/model/ssd_mobilenet_v1_10.onnx --exclude=""
git lfs pull --include=vision/classification/squeezenet/model/squeezenet1.0-9.onnx --exclude=""
git lfs pull --include=vision/body_analysis/emotion_ferplus/model/emotion-ferplus-8.onnx --exclude=""

When running G-API tests, specify OPENCV_GAPI_ONNX_MODEL_PATH environment variable to the model repo root. Expected result:

[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from ONNX
[ RUN      ] ONNX.Infer
[       OK ] ONNX.Infer (37 ms)
[ RUN      ] ONNX.InferROI
[       OK ] ONNX.InferROI (26 ms)
[ RUN      ] ONNX.InferROIList
[       OK ] ONNX.InferROIList (44 ms)
[ RUN      ] ONNX.Infer2ROIList
[       OK ] ONNX.Infer2ROIList (43 ms)
[----------] 4 tests from ONNX (150 ms total)

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

force_builders=Custom
buildworker:Custom=linux-1
build_image:Custom=ubuntu-onnx:20.04
test_modules:Custom=gapi
test_filter:Custom=*ONNX*

CMakeLists.txt

- Basic operations are implemented (Infer, -ROI, -List, -List2); - Implemented automatic preprocessing for ONNX models; - Test suite is extended with `OPENCV_GAPI_ONNX_MODEL_PATH` env for test data (test data is an ONNX Model Zoo repo snapshot); - Fixed kernel lookup logic in core G-API: - Lookup NN kernels not in the default package, but in the associated backend's aux package. Now two NN backends can work in the same graph. - Added Infer SSD demo and a combined ONNX/IE demo;

alalek · 2020-11-03T12:21:10Z

cmake/FindONNX.cmake

@@ -0,0 +1,31 @@
+ocv_clear_vars(HAVE_ONNX)
+
+set(ORT_INSTALL_DIR "" CACHE PATH "ONNX Runtime install directory")


INSTALL_DIR is usually used to specify location where to install something.

Consider renaming to ONNX_ROOT_DIR (do not use ONNX_DIR / ONNX_ROOT as they are reserved by CMake).

Should I change it right now or after having a custom builder working?

Added extra variable into custom builder:

-DONNXRUNTIME_ROOT_DIR=/opt/onnxrt

cmake/FindONNX.cmake

alalek

There is still confusion with ONNX / ONNXRUNTIME naming:

WITH_ONNX
HAVE_ONNX
cmake/FindONNX.cmake (not the same as find_package(ONNX))

alalek · 2020-11-03T12:29:08Z

cmake/FindONNX.cmake

+  set(HAVE_ONNX TRUE)
+  # For CMake output only
+  set(ONNX_LIBRARIES "${ORT_LIB}" CACHE STRING "ONNX Runtime libraries")
+  set(ONNX_INCLUDE_DIR "${ORT_INCLUDE}" CACHE STRING "ONNX Runtime include path")


-ONNX_INCLUDE_DIR +ONNX_INCLUDE_DIRS

Also remove CACHE from both vars (not interned to be specified by user due to precondition above).

if I remove those from CACHE, would the "3rd party" thing display them?

alalek · 2020-11-03T12:30:51Z

cmake/FindONNX.cmake

+if(ORT_INSTALL_DIR)
+  find_library(ORT_LIB onnxruntime
+    ${ORT_INSTALL_DIR}/lib
+    CMAKE_FIND_ROOT_PATH_BOTH)


CMAKE_FIND_ROOT_PATH_BOTH

Do we really need that?

Have no idea, tbh. Probably it works even without it.

alalek · 2020-11-03T12:31:57Z

cmake/FindONNX.cmake

+  set(ONNX_INCLUDE_DIR "${ORT_INCLUDE}" CACHE STRING "ONNX Runtime include path")
+
+  # Link target with associated interface headers
+  set(ONNX_LIBRARY "ort" CACHE STRING "ONNX Link Target")


ort

Please don't obfuscate names.

alalek · 2020-11-03T12:32:55Z

cmake/FindONNX.cmake

+  set_target_properties(${ONNX_LIBRARY} PROPERTIES
+                        INTERFACE_INCLUDE_DIRECTORIES ${ORT_INCLUDE}
+                        IMPORTED_LOCATION ${ORT_LIB}
+                        IMPORTED_IMPLIB ${ORT_LIB})


after that

set(ONNX_LIBRARIES "${ONNX_LIBRARY}")

should be applied.

Then CMake output will text about our internal name of this library, but won't show its full actual path

alalek · 2020-11-03T12:33:27Z

modules/gapi/CMakeLists.txt

+  ocv_target_compile_definitions(${the_module} PRIVATE HAVE_ONNX=1)
+  if(TARGET opencv_test_gapi)
+    ocv_target_compile_definitions(opencv_test_gapi PRIVATE HAVE_ONNX=1)
+    ocv_target_link_libraries(opencv_test_gapi PRIVATE ${ONNX_LIBRARY})


-ONNX_LIBRARY +ONNX_LIBRARIES

should be used.

alalek · 2020-11-03T12:45:22Z

cmake/FindONNX.cmake

@@ -0,0 +1,31 @@
+ocv_clear_vars(HAVE_ONNX)
+
+set(ORT_INSTALL_DIR "" CACHE PATH "ONNX Runtime install directory")


Added extra variable into custom builder:

-DONNXRUNTIME_ROOT_DIR=/opt/onnxrt

dmatveev · 2020-11-03T17:43:12Z

@alalek updated

alalek

CMake part looks good to me 👍

dmatveev · 2020-11-03T18:58:36Z

Thanks!

* G-API: Introduce ONNX backend for Inference - Basic operations are implemented (Infer, -ROI, -List, -List2); - Implemented automatic preprocessing for ONNX models; - Test suite is extended with `OPENCV_GAPI_ONNX_MODEL_PATH` env for test data (test data is an ONNX Model Zoo repo snapshot); - Fixed kernel lookup logic in core G-API: - Lookup NN kernels not in the default package, but in the associated backend's aux package. Now two NN backends can work in the same graph. - Added Infer SSD demo and a combined ONNX/IE demo; * G-API/ONNX: Fix some of CMake issues Co-authored-by: Pashchenkov, Maxim <maxim.pashchenkov@intel.com>

dmatveev requested review from dbudniko, TolyaTalamanov and rgarnov November 3, 2020 08:06

dmatveev self-assigned this Nov 3, 2020

dmatveev added the category: g-api / gapi label Nov 3, 2020

dmatveev added this to the 4.5.1 milestone Nov 3, 2020

alalek reviewed Nov 3, 2020

View reviewed changes

CMakeLists.txt Show resolved Hide resolved

dmatveev force-pushed the dm/upstream_onnx branch from d1f665f to c2d983d Compare November 3, 2020 11:36

alalek reviewed Nov 3, 2020

View reviewed changes

G-API/ONNX: Fix some of CMake issues

6242da8

alalek approved these changes Nov 3, 2020

View reviewed changes

alalek merged commit a110ede into opencv:master Nov 3, 2020

alalek mentioned this pull request Nov 27, 2020

Merge 4.x #18951

Merged

JulienMaille mentioned this pull request Jun 1, 2025

Opencv loads models of dynamic input types #19347

Open

4 tasks

		@@ -0,0 +1,31 @@
		ocv_clear_vars(HAVE_ONNX)

		set(ORT_INSTALL_DIR "" CACHE PATH "ONNX Runtime install directory")

Uh oh!

G-API: Introduce ONNX backend for Inference #18716

G-API: Introduce ONNX backend for Inference #18716

Uh oh!

Conversation

dmatveev commented Nov 3, 2020 • edited by alalek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to build:

How to test:

Pull Request Readiness Checklist

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmatveev commented Nov 3, 2020

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

dmatveev commented Nov 3, 2020

Uh oh!

Uh oh!

dmatveev commented Nov 3, 2020 •

edited by alalek

Loading