ONNX: Add support for Resnet_backbone (Torchvision) #16887

ashishkrshrivastava · 2020-03-23T12:58:03Z

Merge with extra: opencv/opencv_extra#734

resolves #16876

opencv_extra=fasterrcnn

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under OpenCV (BSD) License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

This PR adds

Support to resnet_backbone by adding

fusion of Unfused Frozen BatchNorm.
fusion of unfused Upsample with two inputs.

Unfused Version of batchnorm. Latest version of pytorch ( 1.4 )

Below for are node properties of Reshape nodes

Below attached Fused version of Batchnorm to show comparison between unfused and fused batchnorm.
Initializers inputs of Reshape of above unfused subgraph acts same as Initializer inputs of Fused batchnorm.

Initializers are not considered as nodes.

One more example of Initializer is Weight and bias of Convolution node

And Below mentioned Upsample subgraph with two inputs.

force_builders=Custom,Custom Win,Custom Mac
build_image:Custom=ubuntu-openvino-2020.1.0:16.04
build_image:Custom Win=openvino-2020.1.0
build_image:Custom Mac=openvino-2020.1.0
test_modules:Custom=dnn,python2,python3,java
test_modules:Custom Win=dnn,python2,python3,java
test_modules:Custom Mac=dnn,python2,python3,java
buildworker:Custom=linux-1
# disabled due high memory usage: test_opencl:Custom=ON
test_opencl:Custom=OFF
test_bigdata:Custom=1
test_filter:Custom=*

alalek · 2020-03-24T20:50:00Z

@ashishkrshrivastava Could you please rebase opencv_extra patch (to resolve conflict)?

ashishkrshrivastava · 2020-03-25T03:48:25Z

@alalek I was really really busy for last 2 days. I will complete it by today.

modules/dnn/src/graph_simplifier.cpp

modules/dnn/src/onnx/onnx_graph_simplifier.cpp

modules/dnn/src/onnx/onnx_importer.cpp

dkurt · 2020-04-04T17:46:03Z

@ashishkrshrivastava, thanks for keep working on that! I'm wondering if it's possible to make calculations over constants during import and then just extend Resize parsing, what do you think? Because it looks like that we are trying for now very specific thing but can make subgraph to calculate target size. Can you please check which nodes from the following resize subgraph are computed properly in runtime and which are not?

ashishkrshrivastava · 2020-04-06T13:39:08Z

@dkurt, we just need some changes in onnx_importer.cpp in order to run this version of upsample/Resize subgraph. No need of fusion.
The changes are mentioned below:

else if (layer_type == "Resize")
        {
            for (int i = 1; i < node_proto.input_size(); i++)
                CV_Assert(layer_id.find(node_proto.input(i)) == layer_id.end());
            String interp_mode = layerParams.get<String>("coordinate_transformation_mode");
            CV_Assert_N(interp_mode != "tf_crop_and_resize",
                        interp_mode != "tf_half_pixel_for_nn");
            layerParams.set("align_corners", interp_mode == "align_corners");
            Mat shapes = getBlob(node_proto, constBlobs, node_proto.input_size() - 1);
            CV_CheckEQ(shapes.size[0], 4, "");
            CV_CheckEQ(shapes.size[1], 1, "");
            CV_CheckTypeEQ(shapes.depth(), CV_32S, "");
            int height = shapes.at<int>(2);
            int width  = shapes.at<int>(3);
            if (node_proto.input_size() == 3)
            {
                shapeIt = outShapes.find(node_proto.input(0));
                CV_Assert(shapeIt != outShapes.end());
                MatShape scales = shapeIt->second;
                height *= scales[2];
                width  *= scales[3];
            }
            layerParams.set("width", width);
            layerParams.set("height", height);
            if (layerParams.get<String>("mode") == "linear") {
                layerParams.set("mode", interp_mode == "pytorch_half_pixel" ?
                                        "opencv_linear" : "bilinear");
            }
            replaceLayerParam(layerParams, "mode", "interpolation");
        }
        else if (layer_type == "Upsample")
        {
            layerParams.type = "Resize";
            if (layerParams.has("scales"))
            {
                // Pytorch layer
                DictValue scales = layerParams.get("scales");
                CV_Assert(scales.size() == 4);
                layerParams.set("zoom_factor_y", scales.getIntValue(2));
                layerParams.set("zoom_factor_x", scales.getIntValue(3));
            }
            else
            {
                // Caffe2 layer
                replaceLayerParam(layerParams, "height_scale", "zoom_factor_y");
                replaceLayerParam(layerParams, "width_scale", "zoom_factor_x");
            }
            replaceLayerParam(layerParams, "mode", "interpolation");
            if (node_proto.input_size() == 2 || (layerParams.get<String>("interpolation") == "linear" && framework_name == "pytorch")) {
                layerParams.type = "Resize";
                Mat scales = getBlob(node_proto, constBlobs, 1);
                CV_Assert(scales.total() == 4);
                if (layerParams.get<String>("interpolation") == "linear" && framework_name == "pytorch")
                    layerParams.set("interpolation", "opencv_linear");
                layerParams.set("zoom_factor_y", scales.at<float>(2));
                layerParams.set("zoom_factor_x", scales.at<float>(3));
            }
        }

Change 1.

CV_Assert_N(interp_mode != "tf_crop_and_resize",
                        interp_mode != "tf_half_pixel_for_nn");

from

CV_Assert_N(interp_mode != "tf_crop_and_resize", iinterp != "assymetric",
                        interp_mode != "tf_half_pixel_for_nn");

change 2..

if (node_proto.input_size() == 2 || (layerParams.get<String>("interpolation") == "linear" && framework_name == "pytorch")) {
                layerParams.type = "Resize";
                Mat scales = getBlob(node_proto, constBlobs, 1);
                CV_Assert(scales.total() == 4);
                if (layerParams.get<String>("interpolation") == "linear" && framework_name == "pytorch")
                    layerParams.set("interpolation", "opencv_linear");
                layerParams.set("zoom_factor_y", scales.at<float>(2));
                layerParams.set("zoom_factor_x", scales.at<float>(3));
            }

from

if (layerParams.get<String>("interpolation") == "linear" && framework_name == "pytorch") {
                layerParams.type = "Resize";
                Mat scales = getBlob(node_proto, constBlobs, 1);
                CV_Assert(scales.total() == 4);
                layerParams.set("interpolation", "opencv_linear");
                layerParams.set("zoom_factor_y", scales.at<float>(2));
                layerParams.set("zoom_factor_x", scales.at<float>(3));
            }

Can you please check which nodes from the following resize subgraph are computed properly in runtime and which are not?

So each node is working properly but resize and upsample, adding above changes would make that work too.

modules/dnn/src/onnx/onnx_graph_simplifier.cpp

modules/dnn/test/test_onnx_importer.cpp

modules/dnn/src/onnx/onnx_importer.cpp

dkurt · 2020-04-12T05:57:36Z

modules/dnn/src/onnx/onnx_importer.cpp

+        constant_node->add_output(initializer.name());
+        opencv_onnx::AttributeProto* value = constant_node->add_attribute();
+        opencv_onnx::TensorProto* tensor = value->add_tensors();
+        tensor->CopyFrom(initializer);


Let's also remove initialized nodes in the loop after they copied to the constant node

@dkurt If I remove intializers inside this loop. then this function would be useless because it wont be able to fetch initializer data.

opencv/modules/dnn/src/onnx/onnx_importer.cpp

Lines 133 to 147 in c9c6078

std::map<std::string, Mat> ONNXImporter::getGraphTensors(

const opencv_onnx::GraphProto& graph_proto)

{

opencv_onnx::TensorProto tensor_proto;

std::map<std::string, Mat> layers_weights;

for (int i = 0; i < graph_proto.initializer_size(); i++)

{

tensor_proto = graph_proto.initializer(i);

Mat mat = getMatFromTensor(tensor_proto);

releaseONNXTensor(tensor_proto);

layers_weights.insert(std::make_pair(tensor_proto.name(), mat));

}

return layers_weights;

}

which is being called here

opencv/modules/dnn/src/onnx/onnx_importer.cpp

Lines 330 to 334 in c9c6078

addConstantNodesForInitializers(graph_proto);

simplifySubgraphs(graph_proto);

std::map<std::string, Mat> constBlobs = getGraphTensors(graph_proto);

// List of internal blobs shapes.

So we will have to fetch data from constant nodes, means we need to create a function which first go through all of the nodes and fetch data from all constant nodes.
Am I right ?

Let's see, it set_allocated_t will work - we can keep as is.

@dkurt , please have a look at the changes.

@dkurt Setting tensor directly from initializer wasn't working
value->set_allocated_t(initializer);
It was giving me Segmentation core error so I created a new tensor and set that to value attribute. So it is copied tensor.

opencv_onnx::TensorProto* tensor = initializer.New(); tensor->CopyFrom(initializer); value->set_allocated_t(tensor);

tensor->CopyFrom(initializer); releaseONNXTensor(initializer); value->set_allocated_t(tensor);

where releaseONNXTensor is

void releaseONNXTensor(opencv_onnx::TensorProto& tensor_proto) { if (!tensor_proto.raw_data().empty()) { delete tensor_proto.release_raw_data(); } }

I think you meant the same.

Something like that but we have no raw_data in this case

all the initializers have data type as raw_data. I have checked data types.

frozenBatchNorm2d.onnx bias: raw data running_mean: raw data running_var: raw data weight: raw data

upsample_unfused_two_inputs_opset9_torch1.4.onnx conv1.bias: raw data conv1.weight: raw data conv2.bias: raw data conv2.weight: raw data

upsample_unfused_two_inputs_opset11_torch1.4.onnx conv1.bias: raw data conv1.weight: raw data conv2.bias: raw data conv2.weight: raw data

but yeah, we cant judge by looking at just 3 models, Do you think we should modify this function void releaseONNXTensor(opencv_onnx::TensorProto& tensor_proto) for all data_types or a new function should be created to check and delete all data_types and let this function as it is ?

No, that's not necessary. We can just call method clear_initializer.

i think we cannot use
graph_proto.clear_initializer() before calling getGraphTensors(graph_proto).

And if we use it inside addConstantNodesForInitializers(graph_proto); then would get the following error.
error: (-204:Requested object was not found) Blob 23 not found in const blobs in function 'getBlob'

modules/dnn/src/onnx/onnx_graph_simplifier.hpp

modules/dnn/src/onnx/onnx_graph_simplifier.cpp

modules/dnn/src/onnx/onnx_importer.cpp

dkurt · 2020-04-12T13:16:38Z

modules/dnn/src/onnx/onnx_importer.cpp

                layerParams.set("zoom_factor_y", scales.at<float>(2));
                layerParams.set("zoom_factor_x", scales.at<float>(3));
+                if (layerParams.get<String>("mode") == "linear" && framework_name == "pytorch")
+                    layerParams.set("mode", "opencv_linear");


I think that this logic should be independent from scaling source location. Let's move out of if-else scope

modules/dnn/src/onnx/onnx_importer.cpp

dkurt · 2020-04-16T12:08:28Z

Please take a look at merge artifacts for opencv_extra PR: https://github.com/opencv/opencv_extra/pull/734/files#r407673861.

ashishkrshrivastava · 2020-04-21T00:40:54Z

Please take a look at merge artifacts for opencv_extra PR: https://github.com/opencv/opencv_extra/pull/734/files#r407673861.

Done !

JulienMaille · 2020-04-21T20:09:02Z

@ashishkrshrivastava Right now I need to export my Unet/Resnet with opset=9, other wise the Upsample block is split into many cast/concat blocks like described here (which won't load in OpenCV): pytorch/pytorch#27376 (comment)
Does your PR tackles this? Isn't the problem on pytorch's side?

ashishkrshrivastava · 2020-04-21T23:57:54Z

@JulienMaille support for opset 9 and opset 11 for unfused Upsample and Resize are already added. check #16709 . But those subgraphs has scaling for both H, W. The example you shared has only scale for H only. So subgraph is different accordingly. Still in my view that subgraph will also work because in case of no fusion, each node has to work properly as data flows through each node and OpenCV currently supports all those operator nodes. I would love to add subgraph fusion for such subgraphs, if needed. But for now, it will work without fusion.

dkurt

👍 Looks good to merge, thank you!

ashishkrshrivastava force-pushed the fasterrcnn branch 3 times, most recently from b2f1150 to 61a5c96 Compare March 23, 2020 13:30

dkurt reviewed Mar 25, 2020

View reviewed changes

modules/dnn/src/graph_simplifier.cpp Outdated Show resolved Hide resolved

dkurt reviewed Mar 25, 2020

View reviewed changes

modules/dnn/src/onnx/onnx_graph_simplifier.cpp Outdated Show resolved Hide resolved

dkurt reviewed Mar 25, 2020

View reviewed changes

modules/dnn/src/onnx/onnx_importer.cpp Outdated Show resolved Hide resolved

dkurt reviewed Mar 25, 2020

View reviewed changes

modules/dnn/src/onnx/onnx_importer.cpp Outdated Show resolved Hide resolved

ashishkrshrivastava force-pushed the fasterrcnn branch 5 times, most recently from 8c50c73 to 95ab36b Compare March 29, 2020 05:37

ashishkrshrivastava requested a review from dkurt March 31, 2020 11:02

ashishkrshrivastava mentioned this pull request Apr 1, 2020

Loading ONNX model producing error #16876

Closed

4 tasks

ashishkrshrivastava force-pushed the fasterrcnn branch 3 times, most recently from 79e50ad to 84ef9fb Compare April 3, 2020 06:30

dkurt reviewed Apr 3, 2020

View reviewed changes

modules/dnn/src/onnx/onnx_importer.cpp Outdated Show resolved Hide resolved

dkurt reviewed Apr 9, 2020

View reviewed changes

modules/dnn/src/onnx/onnx_graph_simplifier.cpp Show resolved Hide resolved

ashishkrshrivastava force-pushed the fasterrcnn branch 3 times, most recently from eb5b0ee to c9c6078 Compare April 12, 2020 03:58