You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Resolves#23470. Need to have einsum supported, so link #23134 as well. Einsum is extracted into several operators to bypass this issue first. Will support Einsum in another PR.
To build a demo running SAM with dnn, we actually need two models:
SAM encoder, which is basically a ViT.
can be loaded with dnn.
can be inferred with dnn.
SAM, which takes the output of SAM encoder as input.
can be loaded with dnn.
can be inferred with dnn.
Since the current dnn engine does not support dynamic shape, we need to carefully export ONNX SAM with:
@fengyuentau I tried to load one of the models in your shared folder with OpenCV model diagnostics tool and got a lot of errors. Could you take a look on attached log.
Command line: ./opencv_model_diagnostics -m=./sam_encoder_vit_b.onnx
OS: Ubuntu 18.04, default GCC. msa_diagnostics.txt
With this patch, our dnn engine can only parse and infer sam_encoder_vit_b.sim.onnx and sam_vit_b.fixed.nopost.sim.onnx from my shared model list. Others either require extensive supports to different operators or dynamic input shape, which I will do later in separate pull requests.
./opencv_model_diagnostics -m=sam_vit_b.fixed.nopost.sim.onnx
[ERROR:0@0.044] global debug_utils.cpp:74 printMissing DNN: Not supported types:
Type='ai.onnx.Not', affected nodes:
['/Not_output_0']
To be honest, I do not think my patch affects anything related to this node and SAM just works on my side. You can try the following test code instead:
TEST_P(Test_ONNX_layers, FAIR_SAM)
{
std::string model_path = "/absolute_path/to/sam_vit_b.fixed.nopost.sim.onnx";
Net net = readNet(model_path);
std::vector<int> shape_image_embeddings{1, 256, 64, 64};
Mat image_embeddings(shape_image_embeddings, CV_32FC1);
randn(image_embeddings, 0.f, 1.f);
net.setInput(image_embeddings, std::string("image_embeddings"));
std::vector<int> shape_point_coords{1, 5, 2};
Mat point_coords(shape_point_coords, CV_32FC1);
randn(point_coords, 0.f, 1.f);
net.setInput(point_coords, std::string("point_coords"));
std::vector<int> shape_point_labels{1, 5};
Mat point_labels(shape_point_labels, CV_32FC1);
randn(point_labels, 0.f, 1.f);
net.setInput(point_labels, std::string("point_labels"));
std::vector<int> shape_mask_input{1, 1, 256, 256};
Mat mask_input(shape_mask_input, CV_32FC1);
randn(mask_input, 0.f, 1.f);
net.setInput(mask_input, std::string("mask_input"));
std::vector<int> shape_has_mask_input{1};
Mat has_mask_input(shape_has_mask_input, CV_32FC1);
has_mask_input.at<float>(0) = 1;
net.setInput(has_mask_input, std::string("has_mask_input"));
Mat outs = net.forward();
std::cout << outs.size << std::endl;
}
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #23470.
Need to have einsum supported, so link #23134 as well.Einsum is extracted into several operators to bypass this issue first. Will support Einsum in another PR.To build a demo running SAM with dnn, we actually need two models:
Since the current dnn engine does not support dynamic shape, we need to carefully export ONNX SAM with:
onnxsim
.Access the code of exporting ONNX SAM with encoder: https://github.com/fengyuentau/segment-anything
Download the model I carefully exported: https://drive.google.com/drive/folders/110JBApuq0_37C0gTlMzqs9B3var2oCGY
Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
Patch to opencv_extra has the same branch name.