Refactor DNN module to build with cudnn 9 #25412

ZelboK · 2024-04-14T21:11:14Z

A lot of APIs that are currently being used in the dnn module have been removed in cudnn 9. They were deprecated in 8. This PR updates said code accordingly to the newer API.

Some key notes:

This is my first PR. I am new to openCV.
opencv_test_core tests pass
On a 3080, cuda 12.4(should be irrelevant since I didn't build the opencv_modules, gcc 11.4, WSL 2.
For brevity I will avoid including macro code that will allow for older versions of cudnn to build.

I was unable to get the tests working for opencv_test_dnn and opencv_perf_dnn. The errors I get are of the following:

 OpenCV tests: Can't find required data file: dnn/onnx/conformance/node/test_reduce_prod_default_axes_keepdims_example/model.onnx in function 'findData'
" thrown in the test body.

So before I spend more time investigating I was hoping to get a maintainer to point me in the right direction here. I would like to run these tests and confirm things are working as intended. I may have missed some details.

Pull Request Readiness Checklist

relevant issue
(#24983

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

ZelboK · 2024-04-14T21:16:06Z

I ran the precommit hook, but it is highlighting a lot of formatting related problems. Is there a simple fix for this?

asmorkalov · 2024-04-15T06:11:55Z

@ZelboK thanks for the patch. General comments:

Please do not touch original formatting and the code that is not relevant to CuDNN 9.0.
OpenCV uses opencv_extra repository to store test data. See https://github.com/opencv/opencv/wiki/QA_in_OpenCV. Also there is download_models.py script in opencv_extra/testdata/dnn to download 3rdparty models for testing. Please use it to achieve full dnn coverage locally.

ZelboK · 2024-04-16T16:53:54Z

@asmorkalov
opencv_test_dnn
opencv_test_core
opencv_perf_dnn
opencv_sanity_dnn

All pass. Updated to remove the formatting problems.

Also, is there no formatting hook I could run to do all the formatting automatically for me?

asmorkalov · 2024-04-17T12:01:06Z

The PR breaks CuDNN 8 (v 8.7.0, Ubuntu 18.04):

[ 72%] Building CXX object modules/dnn/CMakeFiles/opencv_dnn.dir/src/layers/const_layer.cpp.o
In file included from /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/csl/cudnn.hpp:8:0,
                 from /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../op_cuda.hpp:12,
                 from /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/blank_layer.cpp:43:
/home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/primitives/../csl/cudnn/recurrent.hpp: In constructor ‘cv::dnn::cuda4dnn::csl::cudnn::RNNDescriptor<T>::RNNDescriptor(const cv::dnn::cuda4dnn::csl::cudnn::Handle&, cv::dnn::cuda4dnn::csl::cudnn::RNNDescriptor<T>::RNNMode, int, int, bool, const cv::dnn::cuda4dnn::csl::cudnn::DropoutDescriptor&)’:
/home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/primitives/../csl/cudnn/recurrent.hpp:129:11: error: ‘input_size’ was not declared in this scope
           input_size, hidden_size,
           ^
/home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/csl/cudnn/cudnn.hpp:22:53: note: in definition of macro ‘CUDA4DNN_CHECK_CUDNN’
     ::cv::dnn::cuda4dnn::csl::cudnn::detail::check((call), CV_Func, __FILE__, __LINE__)
                                                     ^~~~
/home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/primitives/../csl/cudnn/recurrent.hpp:129:11: note: suggested alternative: ‘hidden_size’
           input_size, hidden_size,
           ^
/home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/csl/cudnn/cudnn.hpp:22:53: note: in definition of macro ‘CUDA4DNN_CHECK_CUDNN’
     ::cv::dnn::cuda4dnn::csl::cudnn::detail::check((call), CV_Func, __FILE__, __LINE__)
                                                     ^~~~
In file included from /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/primitives/reshape.hpp:12:0,
                 from /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/blank_layer.cpp:49:
/home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/primitives/../csl/tensor_ops.hpp: In constructor ‘cv::dnn::cuda4dnn::csl::LSTM<T>::LSTM(cv::dnn::cuda4dnn::csl::cudnn::Handle, const cv::dnn::cuda4dnn::csl::LSTM<T>::params_type&)’:
/home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/primitives/../csl/tensor_ops.hpp:548:13: error: ‘filterDesc’ was not declared in this scope
             filterDesc = FilterDescriptor(params.weights_shape);
             ^~~~~~~~~~
modules/dnn/CMakeFiles/opencv_dnn.dir/build.make:853: recipe for target 'modules/dnn/CMakeFiles/opencv_dnn.dir/src/layers/blank_layer.cpp.o' failed
make[3]: *** [modules/dnn/CMakeFiles/opencv_dnn.dir/src/layers/blank_layer.cpp.o] Error 1
make[3]: *** Ожидание завершения заданий…
In file included from /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/csl/cudnn.hpp:8:0,
                 from /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../op_cuda.hpp:12,
                 from /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/const_layer.cpp:10:
/home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/primitives/../csl/cudnn/recurrent.hpp: In constructor ‘cv::dnn::cuda4dnn::csl::cudnn::RNNDescriptor<T>::RNNDescriptor(const cv::dnn::cuda4dnn::csl::cudnn::Handle&, cv::dnn::cuda4dnn::csl::cudnn::RNNDescriptor<T>::RNNMode, int, int, bool, const cv::dnn::cuda4dnn::csl::cudnn::DropoutDescriptor&)’:
/home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/primitives/../csl/cudnn/recurrent.hpp:129:11: error: ‘input_size’ was not declared in this scope
           input_size, hidden_size,
           ^
/home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/csl/cudnn/cudnn.hpp:22:53: note: in definition of macro ‘CUDA4DNN_CHECK_CUDNN’
     ::cv::dnn::cuda4dnn::csl::cudnn::detail::check((call), CV_Func, __FILE__, __LINE__)
                                                     ^~~~
/home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/primitives/../csl/cudnn/recurrent.hpp:129:11: note: suggested alternative: ‘hidden_size’
           input_size, hidden_size,
           ^
/home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/csl/cudnn/cudnn.hpp:22:53: note: in definition of macro ‘CUDA4DNN_CHECK_CUDNN’
     ::cv::dnn::cuda4dnn::csl::cudnn::detail::check((call), CV_Func, __FILE__, __LINE__)
                                                     ^~~~
In file included from /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/primitives/const.hpp:12:0,
                 from /home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/const_layer.cpp:23:
/home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/primitives/../csl/tensor_ops.hpp: In constructor ‘cv::dnn::cuda4dnn::csl::LSTM<T>::LSTM(cv::dnn::cuda4dnn::csl::cudnn::Handle, const cv::dnn::cuda4dnn::csl::LSTM<T>::params_type&)’:
/home/alexander/Projects/OpenCV/opencv-master/modules/dnn/src/layers/../cuda4dnn/primitives/../csl/tensor_ops.hpp:548:13: error: ‘filterDesc’ was not declared in this scope
             filterDesc = FilterDescriptor(params.weights_shape);
             ^~~~~~~~~~
modules/dnn/CMakeFiles/opencv_dnn.dir/build.make:881: recipe for target 'modules/dnn/CMakeFiles/opencv_dnn.dir/src/layers/const_layer.cpp.o' failed

johnnynunez · 2024-04-30T15:08:00Z

The PR breaks CuDNN 8 (v 8.7.0, Ubuntu 18.04):

it was because the PR is incomplete.
Some variables not exists as input_size.
This modules need to refactored

…PIs from cudnn 9. Initial work/foundation Formatting fix. format.

johnnynunez · 2024-05-22T09:03:27Z

great job @asmorkalov! I'm waiting to that for build on jetson

asmorkalov · 2024-05-22T09:39:21Z

There is still LSTM test failure.

modules/dnn/src/cuda4dnn/csl/tensor_ops.hpp

modules/dnn/src/cuda4dnn/primitives/recurrent_cells.hpp

Remove print statements.

remove redundant coment

asmorkalov

👍

Remove trailing white space

ZelboK marked this pull request as draft April 14, 2024 21:11

ZelboK marked this pull request as ready for review April 14, 2024 21:16

asmorkalov added category: gpu/cuda (contrib) OpenCV 4.0+: moved to opencv_contrib incomplete category: dnn labels Apr 15, 2024

ZelboK force-pushed the update-cudnn-to-9 branch from 3bcf55c to 7d9844c Compare April 16, 2024 16:53

asmorkalov removed the incomplete label Apr 16, 2024

asmorkalov self-requested a review April 17, 2024 08:33

ZelboK and others added 2 commits May 17, 2024 15:25

Refactor code to use the updated APIs and remove use of all removed A…

fb10c9b

…PIs from cudnn 9. Initial work/foundation Formatting fix. format.

Fixed build issue with cuDNN 9.1.

8eb703b

asmorkalov force-pushed the update-cudnn-to-9 branch from 7d9844c to 8eb703b Compare May 17, 2024 12:45

asmorkalov added 4 commits May 17, 2024 17:00

Added CuDNN version check for cudnnSetRNNDescriptor_v8

b130499

Fixed code formatting.

5b7aca7

Fixed sigsegv in test devSeqLengths Test_Model.TextRecognition/0

b2fda11

Handle reserved space and workspace for cudnn lstm.

e242ecb

Fixed LSTM in cudnn9, added cudnn6|7|8 and cudnn9 branches for LSTM

9aefd48

asmorkalov reviewed May 27, 2024

View reviewed changes

modules/dnn/src/cuda4dnn/csl/tensor_ops.hpp Outdated Show resolved Hide resolved

modules/dnn/src/cuda4dnn/primitives/recurrent_cells.hpp Outdated Show resolved Hide resolved

ZelboK added 2 commits May 27, 2024 09:02

Update tensor_ops.hpp

3f0cf5b

Remove print statements.

Update recurrent_cells.hpp

3147525

remove redundant coment

asmorkalov approved these changes May 27, 2024

View reviewed changes

asmorkalov self-assigned this May 27, 2024

asmorkalov added this to the 4.10.0 milestone May 27, 2024

Update tensor_ops.hpp

4343d8d

Remove trailing white space

asmorkalov merged commit 05e4860 into opencv:4.x May 28, 2024

mshabunin mentioned this pull request Jun 14, 2024

Merge 4.x -> 5.x #25745

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Refactor DNN module to build with cudnn 9 #25412

Refactor DNN module to build with cudnn 9 #25412

Uh oh!

ZelboK commented Apr 14, 2024 •

edited

Loading

Uh oh!

ZelboK commented Apr 14, 2024

Uh oh!

asmorkalov commented Apr 15, 2024

Uh oh!

ZelboK commented Apr 16, 2024 •

edited

Loading

Uh oh!

asmorkalov commented Apr 17, 2024

Uh oh!

johnnynunez commented Apr 30, 2024

Uh oh!

johnnynunez commented May 22, 2024

Uh oh!

asmorkalov commented May 22, 2024

Uh oh!

Uh oh!

Uh oh!

asmorkalov left a comment

Uh oh!

Uh oh!

Uh oh!

Refactor DNN module to build with cudnn 9 #25412

Refactor DNN module to build with cudnn 9 #25412

Uh oh!

Conversation

ZelboK commented Apr 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

ZelboK commented Apr 14, 2024

Uh oh!

asmorkalov commented Apr 15, 2024

Uh oh!

ZelboK commented Apr 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asmorkalov commented Apr 17, 2024

Uh oh!

johnnynunez commented Apr 30, 2024

Uh oh!

johnnynunez commented May 22, 2024

Uh oh!

asmorkalov commented May 22, 2024

Uh oh!

Uh oh!

Uh oh!

asmorkalov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ZelboK commented Apr 14, 2024 •

edited

Loading

ZelboK commented Apr 16, 2024 •

edited

Loading