| CARVIEW |
Navigation Menu
-
-
Notifications
You must be signed in to change notification settings - Fork 56.2k
[GAPI] Support basic inference in OAK backend #21504
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
@TolyaTalamanov @dmatveev @rgarnov fyi. This is just a basic proposal, so there are several major problems with the current design. |
|
Note: nn metadata can be extracted from DAI once the feature is enabled there: luxonis/depthai-core#368 |
|
@TolyaTalamanov @rgarnov please, review |
smirnov-alexey
changed the title
|
Note:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a few questions for now, will add more on goakbackend.cpp later
|
@rgarnov please, take a look |
|
@alalek can we merge this, please? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks ok, but please bear in mind I didn't dive deep into the backend code.
Just a simple question
- Is there any tests we could run with OAK connected?
- What happens on this:
As a user, I am able to write code like this and from the general G-API concept it is perfectly legal. What will happen here at graph compile time? At graph execution time?
cv::GFrame in; cv::GFrame copy = cv::gapi::oak::copy(cv::gapi::oak::copy(cv::gapi::oak::copy(in))); cv::GFrame render = cv::gapi::renderFrame(copy, ...); cv::GMat out = cv::gapi::streaming::BGR(cv::gapi::oak::copy(render));
|
|
||
| @param model Path to model (.blob file) | ||
| */ | ||
| Params(const std::string &model) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should be explicit
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
| * which is specific to OAK models. | ||
| */ | ||
| struct ParamDesc { | ||
| std::string blob_file; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No input/output layer names here? How is this supposed to work? So far you test with 1-in 1-out models only?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently only 1-in 1-out models are supported. Will extend once supported. Although, not sure if this info will be needed from the user since we extract all meta information from the model directly and utilize in the backend
| G_API_OP(GCopy, <GFrame(GFrame)>, "org.opencv.oak.copy") { | ||
| static GFrameDesc outMeta(const GFrameDesc& in) { | ||
| return in; | ||
| } | ||
| }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is a new operation really required?
https://docs.opencv.org/4.5.5/d6/d91/group__gapi__transform.html#ga1dc3b1987d6685689ecc0d590abddba8
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right, this can be refactored. But for now I'd keep Copy as a separate operation in the backend
| "{ detector | | Path to compiled .blob face detector model }" | ||
| "{ duration | 1 | Number of frames to pull from camera and run inference on }"; | ||
|
|
||
| #ifdef HAVE_OAK |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if this #ifdef at a right place, considering you including oak.hpp explicitly above.
If code compiles with NO OAK but WITH those headers included, do we need it here at all?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed
|
|
||
| // Initialize graph structure | ||
| cv::GFrame in; | ||
| cv::GFrame copy = cv::gapi::oak::copy(in); // NV12 transfered to host + passthrough copy for infer |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe this copy should be isp if we want to trick that it is camera
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Didn't get the idea, could you explain in more details, please?
| int main() { | ||
| GAPI_Assert(false && "Built without OAK support"); | ||
| return -1; | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the app code above can compile with no OAK present, I believe we don't need this either
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed
|
Will address open comments in a separate PR. @alalek can we merge this one, please? |
…ities of OAK backend
smirnov-alexey
force-pushed
the
as/oak_infer
branch
from
e1fa522 to
f714de8
Compare
|
@dmatveev answering your questions above:
|
[GAPI] Support basic inference in OAK backend * Combined commit which enables basic inference and other extra capabilities of OAK backend * Remove unnecessary target options from the cmakelist
Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
Patch to opencv_extra has the same branch name.
This PR brings several new features in OAK backend:
ColorCamera. Note, that it still doesn't support all of the DAI parameters, e.g. resolutionsinfer()operation via a kernel in OAK backend. Currently it only acceptsGFrameas input - not sure if other types are applicable since the stream comes from the camera. Meta information is extracted from the DAI library. Preprocessing is currently tied to camera - this will be refactored separately and done as an additional node before the infer kernel (hidden from the user).Copyoperation in the backend. It allows not to split OAK island in two when we want an original stream from the camera or any other node in the island.Copycan be graph's output, passthrough operation (connects two nodes in the island) or both - the behavior in those cases is different (described and implemented in the backend).Copy