CARVIEW |
Navigation Menu
-
-
Notifications
You must be signed in to change notification settings - Fork 56.2k
GAPI: Add OAK backend #20785
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GAPI: Add OAK backend #20785
Conversation
@dmatveev @TolyaTalamanov @rgarnov fyi. It's still WIP though |
@dmatveev @TolyaTalamanov @AsyaPronina @rgarnov please, take a look. This is rather a draft version and still need some polishing even for a hard-coded pipeline. Still, I would appreciate any feedback. Am I moving to the right direction here? |
The issue with corrupted frames still persists though |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good to know it's alive, but it is far even from an experimental backend yet.
What I'd consider as a bare minimum for merge:
- Generic handling of DAI operations via our kernel interface (let it be NOT public for now)
- Generic handling of camera/non-camera case + tests for these two configurations
Inference, CV ops, etc. can be added atop of this baseline and once the fundamental issues (1) and (2) are solved.
namespace detail { | ||
template<> struct CompileArgTag<gapi::oak::ColorCameraParams> { | ||
static const char* tag() { return "gapi.oak.colorCameraParams"; } | ||
}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we have camera here as a distinct object, should its setup be passed as a graph compile arg?
Actually, when the graph is compiled there's no idea whether a file or a camera will be used as input.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd leave it as is for now. Non-camera input will be introduced later in a separate PR
namespace oak { | ||
|
||
enum class OAKFrameFormat { | ||
BGR = 0, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
MediaFrame has these enums already,
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are more color formats supported by OAK, should we stick to BGR and NV12 only?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should extend MediaFrame
enums then. :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Change to cv::MediaFormat
*in = &(videoEnc->input); | ||
*out = &(videoEnc->bitstream); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
but finally it looks here as intended :)
|
||
GAPI_OAK_KERNEL(GOAKSobelXY, cv::gapi::oak::GSobelXY) { | ||
static void put(const std::unique_ptr<dai::Pipeline>& pipeline, | ||
const std::tuple<int, int>& camera_wh, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why do you need this in every kernel?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some of the DAI operations need to set max resolution size
cv::Size(static_cast<int>(oak_frame->getWidth()), | ||
static_cast<int>(oak_frame->getHeight())), | ||
cv::MediaFormat::NV12, | ||
std::move(oak_frame->getData())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
but why still not to move oak_frame
by itself?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about VectorRef
case below? I think we should be consistent here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could you describe what is VectorRef case (why we introduced it here) and why MediaFrame is not enough?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We use VectorRef
for encoding - it's operation's output. VectorRef
is basically a vector while MediaFrame
is more flexible type. This logic could also be extended in the future for other types like cv::Mat
, Opaque
, etc
size_t out_counter = 0; | ||
for (const auto& d : outs_data) | ||
{ | ||
for (const auto& nh : nodes) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And here you can look at d->inNodes()
(should be only one producer)
fe682dd
to
c2d6fb4
Compare
@dmatveev @sivanov-work please, take a look |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
cv::Size(static_cast<int>(oak_frame->getWidth()), | ||
static_cast<int>(oak_frame->getHeight())), | ||
cv::MediaFormat::NV12, | ||
std::move(oak_frame->getData())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could you describe what is VectorRef case (why we introduced it here) and why MediaFrame is not enough?
cv::MediaFrame::View::Strides{static_cast<long unsigned int>(m_sz.width), | ||
static_cast<long unsigned int>(m_sz.height)}}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Stride is a distance between every row in a plane, in bytes.
If passing width
looks legit here for Y plane, how height
could become a stride for UV
plane?
âš @sivanov-work @rgarnov am I wrong here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That worked somehow, but you're right. Changed to width
and width
for both planes
m_priv(new OAKMediaAdapter::Priv(sz, fmt, y_ptr, uv_ptr)) {}; | ||
|
||
MediaFrame::View OAKMediaAdapter::access(MediaFrame::Access access) { | ||
return m_priv->access(access); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Its debatable but so far OAKMediaAdapter itself may be private so no extra PIMPLing required.
|
||
// 2. Link output nodes to XLinkOut nodes | ||
size_t out_counter = 0; | ||
for (const auto& nh : out_nodes) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there's no strict defined order in out_nodes
right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's aligned with the backend's out_data
and is being used in that aligned way
// 3. Link internal nodes to their parents | ||
for (const auto& nh : inter_nodes) { | ||
// Input nodes in OAK doesn't have parent operation - just camera (for now) | ||
if (std::find(in_nodes.begin(), in_nodes.end(), nh) == in_nodes.end()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This search is linear too. Also why do you need this check here at all? Is your inter_nodes
incorrect?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe it is ok to merge now
GAPI: Add OAK backend * Initial tests and cmake integration * Add a public header and change tests * Stub initial empty template for the OAK backend * WIP * WIP * WIP * WIP * Runtime dai hang debug * Refactoring * Fix hang and debug frame data * Fix frame size * Fix data size issue * Move test code to sample * tmp refactoring * WIP: Code refactoring except for the backend * WIP: Add non-camera sample * Fix samples * Backend refactoring wip * Backend rework wip * Backend rework wip * Remove mat encoder * Fix namespace * Minor backend fixes * Fix hetero sample and refactor backend * Change linking logic in the backend * Fix oak sample * Fix working with ins/outs in OAK island * Trying to fix nv12 problem * Make both samples work * Small refactoring * Remove meta args * WIP refactoring kernel API * Change in/out args API for kernels * Fix build * Fix cmake warning * Partially address review comments * Partially address review comments * Address remaining comments * Add memory ownership * Change pointer-to-pointer to reference-to-pointer * Remove unnecessary reference wrappers * Apply review comments * Check that graph contains only one OAK island * Minor refactoring * Address review comments
Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
Patch to opencv_extra has the same branch name.