CARVIEW |
Navigation Menu
-
-
Notifications
You must be signed in to change notification settings - Fork 56.2k
G-API: Add VPL/VPP preproc core module #21618
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Values(files[0], | ||
files[1], | ||
files[2])); | ||
|
||
using resize_t = std::pair<uint16_t, uint16_t>; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually we always use cv::Size
for that purposes since pair
isn't representative
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok
std::declval<std::tuple<resize_t>>())); | ||
class OneVPLSourcePerf_PP_Test : public TestPerfParams<source_description_preproc_t> {}; | ||
|
||
PERF_TEST_P_(OneVPLSourcePerf_PP_Test, TestPerformance) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This test isn't related to preprocessing
, is it? Why it's in this PR then?
Honestly I didn't get the point what it actually measures here, is it resize + crop?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These tests are performance suite: it measures absolute value of execution time for different kinds of flow:
-source decode by itself
-source decode+processing (transcode)
-source decode + PPEngine (resize)
-source decode + PPEngine in bypass mode (measure pass-through PP const)
These absolute values manual comparison represents relative performance cost of components invocation
modules/gapi/perf/streaming/gapi_streaming_source_perf_tests.cpp
Outdated
Show resolved
Hide resolved
modules/gapi/perf/streaming/gapi_streaming_source_perf_tests.cpp
Outdated
Show resolved
Hide resolved
modules/gapi/perf/streaming/gapi_streaming_source_perf_tests.cpp
Outdated
Show resolved
Hide resolved
@@ -283,12 +293,24 @@ mfxStatus VPLDX11AccelerationPolicy::on_alloc(const mfxFrameAllocRequest *reques | |||
desc.BindFlags = 0; | |||
} | |||
|
|||
/* NB: | |||
* From one hand current OpenVINO API doesn't support texture array and |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for that:
On the one hand
/On the other hand
modules/gapi/src/streaming/onevpl/accelerators/surface/base_frame_adapter.hpp
Outdated
Show resolved
Hide resolved
#include "streaming/onevpl/accelerators/accel_policy_dx11.hpp" | ||
#include "streaming/onevpl/accelerators/dx11_alloc_resource.hpp" | ||
#include "streaming/onevpl/accelerators/utils/shared_lock.hpp" | ||
#define private public |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
????
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
allows to invoke private method of testing component in UTs
@@ -572,9 +572,10 @@ static void setBlob(InferenceEngine::InferRequest& req, | |||
static void setROIBlob(InferenceEngine::InferRequest& req, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BTW, why it's handled only for ROI
? I believe our main use case when:
- IE device -
GPU
- User passed remote context
- MediaFrame comes with
VPL
adapter
In that case we have to do preprocessing on our end since IE
isn't able to do it for us...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In not-long time ago there was simple function SetBlob
in a way SetBlob(...make_shared_blob(Blob, roi)...)
which actually prepare ROI Blob from it's argument - which doesn't work in case of GPU blob
Now SetROIBlob
take care about decision: prepare ROI blob (legacy way) or not (GPU approach)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some small comments from the previous review
modules/gapi/src/streaming/onevpl/engine/preproc/preproc_engine.cpp
Outdated
Show resolved
Hide resolved
modules/gapi/src/streaming/onevpl/engine/preproc/preproc_engine.cpp
Outdated
Show resolved
Hide resolved
modules/gapi/src/streaming/onevpl/engine/preproc/preproc_engine.cpp
Outdated
Show resolved
Hide resolved
modules/gapi/src/streaming/onevpl/engine/preproc/preproc_engine.cpp
Outdated
Show resolved
Hide resolved
modules/gapi/src/streaming/onevpl/engine/preproc/preproc_session.hpp
Outdated
Show resolved
Hide resolved
modules/gapi/src/streaming/onevpl/engine/preproc/preproc_engine.cpp
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you!
modules/gapi/src/streaming/onevpl/accelerators/accel_policy_dx11.cpp
Outdated
Show resolved
Hide resolved
modules/gapi/src/streaming/onevpl/accelerators/accel_policy_dx11.cpp
Outdated
Show resolved
Hide resolved
modules/gapi/src/streaming/onevpl/engine/preproc/preproc_session.hpp
Outdated
Show resolved
Hide resolved
session_type::op_handle_t in_preproc_request {nullptr, | ||
vpl_adapter->get_surface()->get_handle()}; | ||
s->sync_in_queue.emplace(in_preproc_request); | ||
remember_decode_frame(vpl_adapter->get_surface()->get_handle(), in_frame); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we remember this frame somewhere in the process()
(pipeline
's lambdas)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i don't think about that..
But as far i remember pipeline
-lambdas was designed as pure- and is cutting from the other world and cannot receive different kind of data across session
actually it can be passed as another aggregated member of session_type::op_handle_t
as well.
Moreover a subsequent PR about ROI extention follows this approach: https://github.com/opencv/opencv/pull/21554/files#diff-d281efabce2ff069582bd99d9c40e910c4386b65d60010ed9e4b106f4f2b237bR384
@rgarnov Let's move Frame into op_handle_t, what do you think ?
modules/gapi/test/streaming/gapi_streaming_vpp_preproc_test.cpp
Outdated
Show resolved
Hide resolved
modules/gapi/test/streaming/gapi_streaming_vpp_preproc_test.cpp
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No major objections here, but 43 files and around 2K LOC to trigger a simple "resize" via VPL still looks way too much. As now it only can do NV12 Resize isn't it?
} else if ((name == CfgParam::vpp_in_width_name()) || (name == CfgParam::vpp_in_height_name()) || | ||
(name == CfgParam::vpp_in_crop_w_name()) || (name == CfgParam::vpp_in_crop_h_name()) || | ||
(name == CfgParam::vpp_in_crop_x_name()) || (name == CfgParam::vpp_in_crop_y_name()) || | ||
(name == CfgParam::vpp_out_chroma_format_name()) || | ||
(name == CfgParam::vpp_out_width_name()) || (name == CfgParam::vpp_out_height_name()) || | ||
(name == CfgParam::vpp_out_crop_w_name()) || (name == CfgParam::vpp_out_crop_h_name()) || | ||
(name == CfgParam::vpp_out_crop_x_name()) || (name == CfgParam::vpp_out_crop_y_name()) || | ||
(name == CfgParam::vpp_out_pic_struct_name())) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
uh can this be done via some map lookup?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
anyway it is better now than it used to be
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i'm afraid it would require to introduce std::map with lambdas and to fill that lambdas in the same way as this huge if-else implemented with the same amount of conditions
bool FrameInfoComparator::operator()(const mfxFrameInfo& lhs, const mfxFrameInfo& rhs) const { | ||
return lhs < rhs; | ||
} | ||
|
||
bool FrameInfoComparator::equal_to(const mfxFrameInfo& lhs, const mfxFrameInfo& rhs) { | ||
return lhs == rhs; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need this, given that <
and ==
are available already?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in different namespaces:
std::less
which is required for std::map
try to find operator< in global or std namespace and ignored cv::gapi::wip::onevpl. By this reason i use FrameInfoComparator
in std::map
On the other side my UnitTests on preproc requires checking FrameInf
o before and after preproc, which means i should make operator<, and operator== for FrameInfo visible as public symbols in utils.hpp
which looks confusing (for me) because utils is not part of interface
On the other side, because VPPPreprocEngine
is already visible (for UT) purposes and it still require operator< (because ADL doesn't work) then i put aditional operation equal_to
for this comparator class and made the whole file with VPPPreprocEngine visible for UTs
mfxVPPParams.vpp.Out.FourCC = MFX_FOURCC_NV12; | ||
break; | ||
default: | ||
GAPI_LOG_WARNING(nullptr, "Unsupported MediaFormat in preprocessing: " << |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it should be CRITICAL not WARNING, isn't it?
I believe @TolyaTalamanov has added it recently
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Then It requires to rearrange all logger severities in namespace onevpl ( warning ->error/critical). I would implement it in the next PR including other vpl components - not preproc only
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cv::MediaFrame run_sync(const pp_session &session_handle, | ||
const cv::MediaFrame& in_frame) override; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It produces a MediaFrame as output? What's the reason behind this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it emphasizes that it produces new MediaFrame with surface from different pool (not from decoding one).
In bypass-mode (when input & output params are the same) then it is the same in_frame
Since method is called as run_sync
then I suppose it must return result
P.S. It is possible to "upgrade" method for copy-elision and accept R-Value here. Something in a way:
std::string to_lower(std::string &&str)
{
...
return std::move(str);
}
#include <opencv2/gapi/media.hpp> | ||
#include <opencv2/gapi/util/optional.hpp> | ||
|
||
#include "streaming\onevpl\engine\preproc_defines.hpp" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice catch! it's strange but no compiles error
Aha... it's been built for Windows only in CI by now
@@ -47,7 +51,7 @@ enum { | |||
}; | |||
|
|||
GSource::Priv::Priv() : | |||
mfx_handle(MFXLoad()), | |||
// mfx_handle(MFXLoad()), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
???
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
mfx_handle
is a global variable by now. It was implemented to overcome oneVPL issue with cloning session:
all VPL operations (decode, VPP) must belong to mfxSession
context. this mfxSession holds DevicePtr, bitstream settings and some other infrastructural data. So to queue VPP preprocessing operation we must create different mfxSession
wich will share some settings with decoding mfxSession
and has different another's. At least both session must share the same VPL Library Implementation and version which selected in VPLSource configuration step.
The ideas was:
pass decode session object as MediaFrameAdapter data, then PreprocEngine extract it from MediaFrameAdapter in is_applicable
method and perform mfxSessionClone
in order to share the MAJOR session params: Implementation, Version, Decoder ID and so; then operates with VPP cloned session and forget about decode session (to avoid huge operation for mfxLoad and choosing implementation based on selected criteria, which may be diffrent from producing VPL source and so on).
But i faced with some issue in oneVPL implementation which rejects cloning session due to some inner bug.
To overcome this issue i made mfx_handle
as global variable (temporary) with limitation "only single VPL Source can exist" then i just use this global mfx_handle
state to create a NEW session (not as clone) from it's state.
After new VPL version releases we will reactivate idea with Cloning Session and revert such commented lines back
@@ -261,7 +266,7 @@ GSource::Priv::~Priv() { | |||
GAPI_LOG_INFO(nullptr, "Unload MFX implementation description: " << mfx_impl_description); | |||
MFXDispReleaseImplDescription(mfx_handle, mfx_impl_description); | |||
GAPI_LOG_INFO(nullptr, "Unload MFX handle: " << mfx_handle); | |||
MFXUnload(mfx_handle); | |||
//MFXUnload(mfx_handle); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
???
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A small comment, please check but only if there would be some other commits required
@alalek could you merge it please? |
G-API: Add VPL/VPP preproc core module * Add BaseMediAdapter for VPL * Add PreprocSession & PreprocEngine interface part * Implement preproc UT, Fix base path * Add common PP interface, add common pp_params * Rough decoupling VPL & Preproc * Add syntax sugar for PP interface * Integrate VPP preproc in GIEbackend * Add PP bypass * Add perf tests for PP * Fix warning in vpl core UT * Add inner preproc resolution Unit Test * Remove VPP preproc description from single ROI sample * Apply SetROIBlob for diferent Infer operations * Eliminate extra branch-lines for cfg_param_parser & transcode_engine * Fix UT warning &PreprocSession compile * Fix compilation & warnings * Reduce Session&Engine code amount * Apply some comments * Revert IE changes, rename preproc * Fix for DX11 infer for OV: turn off texture array * Remove dependency PP on IE * Change fixture tests params * Apply other comments & turn off ROI for GPU * Fix compilation: remove forgotten INFER define * Apply debt comments * Fix PP UTs: add FrameInfo value comparator * Fix style * Remove standalone map for preproc frames storage * Add other comments
Core part of #21363
VPP preproc by itself with applied comments
Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
Patch to opencv_extra has the same branch name.
Build Configuration