G-API: Refine Semantic Segmentation Demo #23766

TolyaTalamanov · 2023-06-09T07:29:01Z

Overview

Supported demo working with camera id (e.g --input=0)
Supported 3d output segmentation models (e.g deeplabv3)
Supported desync execution
Supported higher camera resolution
Changed the color map to pascal voc (https://cloud.githubusercontent.com/assets/4503207/17803328/1006ca80-65f6-11e6-9ff6-36b7ef5b9ac6.png)

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

TolyaTalamanov · 2023-06-09T07:30:09Z

modules/gapi/include/opencv2/gapi/streaming/cap.hpp

-    explicit GCaptureSource(int id) : cap(id) { prep(); }
-    explicit GCaptureSource(const std::string &path) : cap(path) { prep(); }
+    using Properties = std::unordered_map<int, double>;
+    explicit GCaptureSource(int id, const Properties &properties = {})


Well, I wish it was separate method set but properties must be set before the first frame is read (prep())

TolyaTalamanov · 2023-06-09T07:31:28Z

modules/gapi/samples/semantic_segmentation.cpp

+}
+
+static void drawResults(cv::Mat &img, const cv::Mat &color_mask) {
+    img = img / 2 + color_mask / 2;


Not so efficient due reallocations but it doesn't impact performance

Probably we should thing how we can make stuff like this in graph -- where a desynchronized path meets the synchronized one, and results are accumulated inside as we do now manually..

Good idea for future versions

TolyaTalamanov · 2023-06-09T07:32:13Z

modules/gapi/samples/semantic_segmentation.cpp

        cv::Mat mask_img;
        classesToColors(classes, mask_img);
-        cv::resize(mask_img, out, in.size());
+        cv::resize(mask_img, out, in.size(), 0, 0, cv::INTER_NEAREST);


INTER_NEAREST is better when it comes to resize color mask

TolyaTalamanov · 2023-06-09T07:32:40Z

modules/gapi/samples/semantic_segmentation.cpp

+        source = std::make_shared<cv::gapi::wip::GCaptureSource>(
+            std::stoi(input),
+            cv::gapi::wip::GCaptureSource::Properties {
+              {cv::CAP_PROP_FRAME_WIDTH, 1280},


Hardcoded so far, not sure if it should be cmd option

TolyaTalamanov · 2023-06-09T07:37:03Z

@dmatveev Could you have a look, please?

TolyaTalamanov · 2023-06-09T08:44:58Z

modules/gapi/include/opencv2/gapi/streaming/cap.hpp

-    explicit GCaptureSource(int id) : cap(id) { prep(); }
-    explicit GCaptureSource(const std::string &path) : cap(path) { prep(); }
+    using Properties = std::unordered_map<int, double>;
+    explicit GCaptureSource(int id, const Properties &properties = {})


Not exposed to python so far...

Exposed now

dmatveev · 2023-06-09T08:45:40Z

modules/gapi/samples/semantic_segmentation.cpp

+static std::vector<cv::Vec3b> colors = {
+    {0, 0, 0},       {0, 0, 128},     {0, 128, 0}, {0, 128, 128}, {128, 0, 0},   {128, 0, 128}, {128, 128, 0},
+    {128, 128, 128}, {0, 0, 64},      {0, 0, 192}, {0, 128, 64},  {0, 128, 192}, {128, 0, 64},  {128, 0, 192},
+    {128, 128, 64},  {128, 128, 192}, {0, 64, 0},  {0, 64, 128},  {0, 192, 0},   {0, 192, 128}, {128, 64, 0}
 };


What's the reason for this change? Previously it was just one line per class, easy to edit.

No problem, I'll keep old alignment

dmatveev · 2023-06-09T08:47:46Z

modules/gapi/samples/semantic_segmentation.cpp

+}
+
+static void drawResults(cv::Mat &img, const cv::Mat &color_mask) {
+    img = img / 2 + color_mask / 2;


Probably we should thing how we can make stuff like this in graph -- where a desynchronized path meets the synchronized one, and results are accumulated inside as we do now manually..

Good idea for future versions

TolyaTalamanov · 2023-06-12T11:46:44Z

@dmatveev @asmorkalov Hi folks, can we merge this?

asmorkalov · 2023-06-13T06:32:51Z

modules/gapi/samples/semantic_segmentation.cpp

+              {cv::CAP_PROP_FRAME_HEIGHT, 720},
+              {cv::CAP_PROP_BUFFERSIZE, 1},
+              {cv::CAP_PROP_AUTOFOCUS, true},
+              {cv::CAP_PROP_FOURCC, cv::VideoWriter::fourcc('M', 'J', 'P', 'G')}


It works not with all cameras. I propose to add note on required camera parameters to help message and the source file header.

Good point, I'd rather remove it then

modules/python/src2/cv2_convert.hpp

…aTalamanov/opencv into at/segmentation-demo-desync

asmorkalov

👍

…emo-desync G-API: Refine Semantic Segmentation Demo opencv#23766 ### Overview * Supported demo working with camera id (e.g `--input=0`) * Supported 3d output segmentation models (e.g `deeplabv3`) * Supported `desync` execution * Supported higher camera resolution * Changed the color map to pascal voc (https://cloud.githubusercontent.com/assets/4503207/17803328/1006ca80-65f6-11e6-9ff6-36b7ef5b9ac6.png) ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [ ] I agree to contribute to the project under Apache 2 License. - [ ] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [ ] The PR is proposed to the proper branch - [ ] There is a reference to the original bug report and related work - [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [ ] The feature is well documented and sample code can be built with the project CMake

TolyaTalamanov added 7 commits June 7, 2023 10:21

Add desync and fps

7d8de4d

Fix OV backend header guards

2591813

Change color pallette

02f327d

Add camera prop

4b9c82d

Add more camera properties

8c6fb65

Fix desync usage

4e903ab

Support properties for source and fix demo

329fabd

TolyaTalamanov commented Jun 9, 2023

View reviewed changes

Refine segmetnation demo

cf2505c

TolyaTalamanov force-pushed the at/segmentation-demo-desync branch from a19aa1c to cf2505c Compare June 9, 2023 07:36

asmorkalov requested a review from dmatveev June 9, 2023 07:58

asmorkalov added the category: g-api / gapi label Jun 9, 2023

asmorkalov assigned dmatveev Jun 9, 2023

TolyaTalamanov commented Jun 9, 2023

View reviewed changes

dmatveev approved these changes Jun 9, 2023

View reviewed changes

TolyaTalamanov added 3 commits June 12, 2023 09:05

Fix color map formating

4d123fb

Wrap capture source properties into python

cf8dc59

Add map header

a9c6527

asmorkalov reviewed Jun 13, 2023

View reviewed changes

TolyaTalamanov added 2 commits June 13, 2023 09:40

Merge branch 'at/segmentation-demo-desync' of https://github.com/Toly…

0ec5a36

…aTalamanov/opencv into at/segmentation-demo-desync

Remove explicit codec

967ff08

asmorkalov approved these changes Jun 13, 2023

View reviewed changes

asmorkalov added this to the 4.8.0 milestone Jun 13, 2023

asmorkalov merged commit a371bda into opencv:4.x Jun 13, 2023

asmorkalov mentioned this pull request Jul 12, 2023

(5.x) Merge 4.x #23979

Merged

Uh oh!

G-API: Refine Semantic Segmentation Demo #23766

G-API: Refine Semantic Segmentation Demo #23766

Uh oh!

Conversation

TolyaTalamanov commented Jun 9, 2023

Overview

Pull Request Readiness Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TolyaTalamanov commented Jun 9, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TolyaTalamanov commented Jun 12, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

asmorkalov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!