Open VideoCapture from data stream #25584

dkurt · 2024-05-14T12:26:25Z

Pull Request Readiness Checklist

Add VideoCapture option to read a raw binary video data from std::streambuf.

There are multiple motivations:

Avoid disk file creation in case of video already in memory (received by network or from database).
Streaming mode. Frames decoding starts during sequential file transfer by chunks.

Suppoted backends:

FFmpeg
MSMF (no streaming mode)

Supporter interfaces:

C++ (std::streambuf)
Python (io.BufferedIOBase)

resolves #24400

test h264
test IP camera like approach with no metadata but key frame only?
C API plugin

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

opencv-alalek

Plugin support is necessary - it is mandatory for FFmpeg support on Windows.

modules/videoio/include/opencv2/videoio.hpp

modules/videoio/src/plugin_capture_api.hpp

dkurt · 2024-06-07T07:09:55Z

Discussed on OpenCV weekly sync about potential improvements:

On current PR state, do not use std::vector but allow user pass just pointer/buffer_size.
Consider alternative API with a kind of streaming by chunks. Might be more efficient with GPU decoders.

mshabunin · 2024-06-07T08:09:53Z

We have something similar in cudacodec module:

RawVideoSource object is an iterator over encoded data stream (https://github.com/opencv/opencv_contrib/blob/b236c71c2f8d983403c35a0cea8bec0432a4b0fe/modules/cudacodec/include/opencv2/cudacodec.hpp#L508-L555) and can be used to construct VideoReader (https://github.com/opencv/opencv_contrib/blob/b236c71c2f8d983403c35a0cea8bec0432a4b0fe/modules/cudacodec/include/opencv2/cudacodec.hpp#L604).

Following example in the test decodes key frames (if I understand corretly): https://github.com/opencv/opencv_contrib/blob/b236c71c2f8d983403c35a0cea8bec0432a4b0fe/modules/cudacodec/test/test_video.cpp#L178-L209

gdippolito · 2024-08-14T10:01:07Z

Hello, what is left to complete this work? Would anyone be able to complete this work by any chance?

dkurt · 2024-08-16T07:41:42Z

@gdippolito, I can complete this one once I can figure out which API is better. Last time we discussed in the team, more generic is to introduce kind of streaming API which returns chunks of video buffer.

gdippolito · 2024-10-20T17:40:46Z

Hey @dkurt thanks for your work on this PR. Do you think this is ready for some testing?

dkurt · 2024-10-21T05:43:25Z

@gdippolito, thanks! Yes, it's ready and any feedback is welcome.

dkurt · 2024-12-24T07:27:01Z

std::streambuf is the I/O stream object at first (but "buffered" variant).
They are not buffers.

Totally fine to keep stream notation everywhere. There is just a conflict in public API with filename (getStreamBackends)

opencv-alalek · 2024-12-24T14:29:00Z

modules/videoio/src/cap_ffmpeg.cpp

+    try
+    {
+        cv::VideoCaptureParameters parameters(params, n_params);
+        cap = new CvCapture_FFMPEG_proxy(makePtr<ReadStreamPluginProvider>(opaque, read, seek), parameters);


makePtr<ReadStreamPluginProvider>(opaque, read, seek)

Released right after this call because it is not saved anywhere.

As a workaround, readStream = stream for now

dkurt · 2024-12-24T18:09:03Z

modules/videoio/include/opencv2/videoio.hpp

@@ -719,6 +719,25 @@ enum VideoCaptureOBSensorProperties{

 //! @} videoio_flags_others

+/** @brief Read data stream interface
+ */
+class CV_EXPORTS_W IReadStream


@opencv-alalek, @asmorkalov , need your opinion on naming as we ready to move it to the public API. There are VideoCapture, IStreamSource , VideoReader, VideoWriter. So does it make sense to keep the order and name it IStreamReader? Moreover, it matches PyTorch's naming: https://pytorch.org/audio/main/generated/torchaudio.io.StreamReader.html#streamreader

IReadStream -> IStreamReader ReadStreamPluginProvider -> PluginStreamReader PythonReadStream -> PythonStreamReader BufferBackend -> StreamBufferedBackend

opencv-alalek · 2024-12-24T19:23:01Z

BTW, there is leftover with Python typing generator (map to io.BufferedIOBase or even io.BufferedReader)

opencv-alalek · 2024-12-25T09:59:13Z

There is problem with iOS/Swift bindings:

<module-includes>:1:9: note: in file included from <module-includes>:1:
#import "Headers/opencv2.h"
        ^
/Users/opencv-cn/GHA-OCV-4/_work/opencv/opencv/ios/build/build-armv7-iphoneos/lib/Release/opencv2.framework/Headers/opencv2.h:151:9: note: in file included from /Users/opencv-cn/GHA-OCV-4/_work/opencv/opencv/ios/build/build-armv7-iphoneos/lib/Release/opencv2.framework/Headers/opencv2.h:151:
#import <opencv2/VideoCapture.h>
        ^
/Users/opencv-cn/GHA-OCV-4/_work/opencv/opencv/ios/build/build-armv7-iphoneos/lib/Release/opencv2.framework/Headers/VideoCapture.h:288:1: error: duplicate declaration of method 'open:apiPreference:params:'
- (BOOL)open:(IStreamReader*)source apiPreference:(int)apiPreference params:(IntVector*)params NS_SWIFT_NAME(open(source:apiPreference:params:));
^
/Users/opencv-cn/GHA-OCV-4/_work/opencv/opencv/ios/build/build-armv7-iphoneos/lib/Release/opencv2.framework/Headers/VideoCapture.h:224:1: note: previous declaration is here
- (BOOL)open:(NSString*)filename apiPreference:(int)apiPreference params:(IntVector*)params NS_SWIFT_NAME(open(filename:apiPreference:params:));
^
<unknown>:0: error: could not build Objective-C module 'opencv2'

Error message is not very clear to me

opencv-alalek · 2024-12-25T10:41:25Z

Looks like fix works: https://github.com/opencv/opencv/actions/runs/12491953263/job/34858615536?pr=25584

@dkurt Feel free to push your changes.

dkurt · 2024-12-25T10:50:08Z

@opencv-alalek no pending changes left

modules/videoio/include/opencv2/videoio.hpp

mshabunin · 2024-12-25T11:46:21Z

modules/videoio/include/opencv2/videoio.hpp

+    /** @brief Sets the stream position
+     *
+     * @param offset Seek offset
+     * @param origin SEEK_SET / SEEK_END / SEEK_CUR


Wouldn't it be better to completely avoid C/C++ constants in public interface and introduce our own?

FFmpeg (basically) and Python use them too (with same values).
IMHO, it is not necessary to introduce extra conversions to/from own values.

Agree, we can add such enum but potential user will still cast to integer or introduce long if/else/switch

My concern was that user would be required to include <stdio.h>/<cstdio> in order to use this seemingly unrelated class. Though perhaps it's fine.

Open VideoCapture from data stream opencv#25584 ### Pull Request Readiness Checklist Add VideoCapture option to read a raw binary video data from `std::streambuf`. There are multiple motivations: 1. Avoid disk file creation in case of video already in memory (received by network or from database). 2. Streaming mode. Frames decoding starts during sequential file transfer by chunks. Suppoted backends: * FFmpeg * MSMF (no streaming mode) Supporter interfaces: * C++ (std::streambuf) * Python (io.BufferedIOBase) resolves opencv#24400 - [x] test h264 - [x] test IP camera like approach with no metadata but key frame only? - [x] C API plugin See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

dkurt added category: videoio feature labels May 14, 2024

dkurt force-pushed the videocapture_from_buffer branch from 2bd1a39 to 023eeb5 Compare May 14, 2024 12:42

dkurt mentioned this pull request May 15, 2024

VideoCapture add support for byte buffer #24400

Closed

dkurt changed the title ~~Open VideoCapture from memory buffer~~ Open VideoCapture from memory buffer (FFmpeg backend only) May 15, 2024

dkurt marked this pull request as ready for review May 15, 2024 14:50

dkurt requested review from asmorkalov, mshabunin and opencv-alalek May 15, 2024 14:51

opencv-alalek added the RFC label May 16, 2024

dkurt changed the title ~~Open VideoCapture from memory buffer (FFmpeg backend only)~~ Open VideoCapture from memory buffer May 16, 2024

dkurt force-pushed the videocapture_from_buffer branch from 4dace6c to b71844d Compare May 16, 2024 19:24

opencv-alalek reviewed May 17, 2024

View reviewed changes

modules/videoio/include/opencv2/videoio.hpp Outdated Show resolved Hide resolved

modules/videoio/include/opencv2/videoio.hpp Outdated Show resolved Hide resolved

asmorkalov added this to the 4.11.0 milestone May 17, 2024

opencv-alalek reviewed May 19, 2024

View reviewed changes

modules/videoio/src/plugin_capture_api.hpp Outdated Show resolved Hide resolved

dkurt force-pushed the videocapture_from_buffer branch from ec90def to eeae54d Compare June 5, 2024 06:04

dkurt marked this pull request as draft June 7, 2024 07:10

dkurt force-pushed the videocapture_from_buffer branch from eeae54d to 175f070 Compare September 28, 2024 07:40

dkurt force-pushed the videocapture_from_buffer branch 3 times, most recently from 214cd8a to 62cd53a Compare October 10, 2024 19:22

dkurt marked this pull request as ready for review October 14, 2024 11:05

dkurt changed the title ~~Open VideoCapture from memory buffer~~ Open VideoCapture from std::streambuf Oct 15, 2024

opencv-alalek self-assigned this Dec 24, 2024

opencv-alalek reviewed Dec 24, 2024

View reviewed changes

opencv-pushbot force-pushed the videocapture_from_buffer branch from ffb04fd to ad2b7b3 Compare December 24, 2024 16:40

videoio: open stream fixes

64a7d96

opencv-pushbot force-pushed the videocapture_from_buffer branch from ad2b7b3 to 64a7d96 Compare December 24, 2024 17:55

dkurt commented Dec 24, 2024

View reviewed changes

Apply renamings:

63b09e1

IReadStream -> IStreamReader ReadStreamPluginProvider -> PluginStreamReader PythonReadStream -> PythonStreamReader BufferBackend -> StreamBufferedBackend

dkurt changed the title ~~Open VideoCapture from std::streambuf~~ Open VideoCapture from data stream Dec 24, 2024

opencv-alalek approved these changes Dec 24, 2024

View reviewed changes

opencv-alalek assigned asmorkalov and unassigned opencv-alalek Dec 24, 2024

Resolve Windows warning

cfed93c

dkurt force-pushed the videocapture_from_buffer branch from 2a02297 to cfed93c Compare December 25, 2024 07:59

try to fix objc bindings

97b0d1a

mshabunin reviewed Dec 25, 2024

View reviewed changes

modules/videoio/include/opencv2/videoio.hpp Outdated Show resolved Hide resolved

mshabunin reviewed Dec 25, 2024

View reviewed changes

Remove std::streambuf overload

f3de4d1

asmorkalov merged commit e9982e8 into opencv:4.x Dec 26, 2024
27 of 31 checks passed

asmorkalov mentioned this pull request Jan 15, 2025

5.x merge 4.x #26768

Merged

dkurt deleted the videocapture_from_buffer branch January 16, 2025 18:12

This was referenced Jan 18, 2025

[Feature Request] Support input via pipe Breakthrough/PySceneDetect#189

Open

Request: Process video stream from stdin Breakthrough/DVR-Scan#137

Open

dkurt mentioned this pull request May 6, 2025

Java VideoCapture buffered stream constructor #27284

Merged

6 tasks

Uh oh!

Open VideoCapture from data stream #25584

Open VideoCapture from data stream #25584

Uh oh!

Conversation

dkurt commented May 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

opencv-alalek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dkurt commented Jun 7, 2024

Uh oh!

mshabunin commented Jun 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gdippolito commented Aug 14, 2024

Uh oh!

dkurt commented Aug 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gdippolito commented Oct 20, 2024

Uh oh!

dkurt commented Oct 21, 2024

Uh oh!

dkurt commented Dec 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

opencv-alalek Dec 24, 2024

Choose a reason for hiding this comment

Uh oh!

dkurt Dec 24, 2024

Choose a reason for hiding this comment

Uh oh!

dkurt Dec 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

opencv-alalek commented Dec 24, 2024

Uh oh!

opencv-alalek commented Dec 25, 2024

Uh oh!

opencv-alalek commented Dec 25, 2024

Uh oh!

dkurt commented Dec 25, 2024

Uh oh!

Uh oh!

mshabunin Dec 25, 2024

Choose a reason for hiding this comment

Uh oh!

opencv-alalek Dec 25, 2024

Choose a reason for hiding this comment

Uh oh!

dkurt Dec 25, 2024

Choose a reason for hiding this comment

Uh oh!

mshabunin Dec 25, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dkurt commented May 14, 2024 •

edited

Loading

mshabunin commented Jun 7, 2024 •

edited

Loading

dkurt commented Aug 16, 2024 •

edited

Loading

dkurt commented Dec 24, 2024 •

edited

Loading

dkurt Dec 24, 2024 •

edited

Loading