TFLite models importer #23161

dkurt · 2023-01-20T14:57:58Z

Pull Request Readiness Checklist

Merge with: opencv/opencv_extra#1042

resolves #13918

How to build (tested on Ubuntu 20.04)

git clone -b v23.1.21 https://github.com/google/flatbuffers
cmake -S flatbuffers -B fbs_build
cmake --build fbs_build -j4
cmake --install fbs_build --prefix fbs_install

export flatbuffers_DIR=$HOME/fbs_install/lib/cmake/flatbuffers/
cmake \
    -DCMAKE_BUILD_TYPE=Release \
    -DBUILD_LIST=ts,dnn,python3 \
    -DWITH_FLATBUFFERS=ON \
    -S opencv -B opencv_build

How to build for OpenCV.js:

emcmake python3 ./opencv/platforms/js/build_js.py build_wasm \
  --build_wasm \
  --cmake_option="-DWITH_FLATBUFFERS=ON" \
  --cmake_option="-Dflatbuffers_DIR=$HOME/fbs_install/lib/cmake/flatbuffers/"

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

force_builders=Linux OpenCL,Linux AVX2
buildworker:Linux OpenCL=linux-1

modules/dnn/src/layers/fast_convolution/fast_convolution.cpp

modules/dnn/test/test_tflite_importer.cpp

cmake/OpenCVFindFlatBuffers.cmake

modules/dnn/include/opencv2/dnn/dnn.hpp

CMakeLists.txt

modules/dnn/src/tflite/schema.fbs

alalek · 2023-02-06T20:04:42Z

modules/dnn/src/tflite/schema.fbs

+// Version 3b: Rename fields in SignatureDef. Has backward compatibility with
+//             version 3 and 3a.
+
+namespace tflite;


Should we expect the similar conflicts like protobuf's caffe?

Replaced to opencv_tflite

alalek · 2023-02-06T20:07:06Z

modules/dnn/misc/tflite/schema_generated.h

+// Ensure the included flatbuffers.h is the same version as when this file was
+// generated, otherwise it may not be compatible.
+static_assert(FLATBUFFERS_VERSION_MAJOR == 23 &&
+              FLATBUFFERS_VERSION_MINOR == 1 &&
+              FLATBUFFERS_VERSION_REVISION == 4,
+             "Non-compatible flatbuffers version included");


There is the same problem with version check in protobuf.

we should have 3rdparty's bundle of flatbuffers of this exact version.

and support re-generation for this file (or create it on the fly)

Yeah, that's a problem. In the latest commit I've removed schema_generated.h and added a command to generate it. Have not tested with Emscripten or cross-compilation yet.

alalek · 2023-02-06T20:11:52Z

modules/dnn/test/test_tflite_importer.cpp

+template<typename TString>
+static std::string _tf(TString filename)
+{
+    return (getOpenCVExtraDir() + "/dnn/tflite/") + filename;


We don't need this helper.
findDataFile() should be used instead.

The same note is about getOpenCVExtraDir() usage.

We don't need this wrapper.
But we need "required" parameter which should be false for models from download_models.py.
It is better to use findDataFile() directly.

Sure, done.

modules/dnn/src/tflite/tflite_importer.cpp

fengyuentau · 2023-02-08T16:02:33Z

CMakeLists.txt

@@ -471,6 +471,9 @@ OCV_OPTION(WITH_OBSENSOR "Include obsensor support (Orbbec RGB-D modules: Astra+
 OCV_OPTION(WITH_CANN "Include CANN support" OFF
  VISIBLE_IF TRUE
  VERIFY HAVE_CANN)
+OCV_OPTION(WITH_FLATBUFFERS "Include FlatBuffers support" OFF


Have a concern here: people would not know they need to turn it on for the support of TFLite models, because there are no connections in terms of their names.

Also is it possible to bring flatbuffers into 3rdparty like protobuf so that we can distribute pre-built packages with this support?

I added a corresponding exception message when user tries load TFLite model. For Caffe, TF and ONNX the same flag is WITH_PROTOBUF (however enabled by default).

Also is it possible to bring flatbuffers into 3rdparty like protobuf so that we can distribute pre-built packages with this support?

It's possible but I don't know if it's suitable to add a new 3rdparty dependency just for TFLite.

@dkurt I believe we could start with the current state and add 3rdparty later as a separate activity.
Could you please add information into PR's description about required build environment configuration:

Ubuntu version, used packages names and versions?

what is needed to build OpenCV.js through Emscripten?

I added a corresponding exception message when user tries load TFLite model.

Okay, that makes sense. Adding as 3rdparty can be done later.

@alalek, added, thanks! Emscripten command is turned pretty smooth.

fengyuentau · 2023-02-08T16:17:16Z

modules/dnn/src/tflite/tflite_importer.cpp

+    layerParams.set("pool_pad_h", 0);
+}
+
+void TFLiteImporter::parseReshape(const Operator& op, const std::string& opcode, LayerParams& layerParams) {


Not familiar with TFLite models but do we need to consider constant folding for some of the operators?

Not sure that it must be done at import step due DNN has own fusion at the model initialization. If you mean something similar to TensorFlow/ONNX graph ops fusion, TFLite basic layers (see the schema) are simple enough and mapped 1:1 to OpenCV layers. At least according the models I tried from Mediapipe.

If you mean something similar to TensorFlow/ONNX graph ops fusion

Yes, I meant this. Just wonder whether there are similar cases in TFLite like "[Input]->Identity->..." where the Identity can be eliminated in ONNX. But still this can be a further optimization if we comes across these cases.

Well, yes, there are builtin operations which probably can be a part of single layers. I had an idea to make a something like general rules between TensorFlow and TFLite importers for NHWC-NCHW layout tracking and subgraph fusion. However, it may be also done later with more TFLite models examples.

From schema.fbs

enum BuiltinOperator : int32 { ADD = 0, AVERAGE_POOL_2D = 1, CONCATENATION = 2, CONV_2D = 3, DEPTHWISE_CONV_2D = 4, DEPTH_TO_SPACE = 5, DEQUANTIZE = 6, EMBEDDING_LOOKUP = 7, FLOOR = 8, FULLY_CONNECTED = 9, HASHTABLE_LOOKUP = 10, L2_NORMALIZATION = 11, L2_POOL_2D = 12, LOCAL_RESPONSE_NORMALIZATION = 13, LOGISTIC = 14, LSH_PROJECTION = 15, LSTM = 16, MAX_POOL_2D = 17, MUL = 18, RELU = 19, // NOTE(aselle): RELU_N1_TO_1 used to be called RELU1, but it was renamed // since different model developers use RELU1 in different ways. Never // create another op called RELU1. RELU_N1_TO_1 = 20, RELU6 = 21, RESHAPE = 22, RESIZE_BILINEAR = 23, RNN = 24, SOFTMAX = 25, SPACE_TO_DEPTH = 26, SVDF = 27, TANH = 28, CONCAT_EMBEDDINGS = 29, SKIP_GRAM = 30, CALL = 31, CUSTOM = 32, EMBEDDING_LOOKUP_SPARSE = 33, PAD = 34, UNIDIRECTIONAL_SEQUENCE_RNN = 35, GATHER = 36, BATCH_TO_SPACE_ND = 37, SPACE_TO_BATCH_ND = 38, TRANSPOSE = 39, MEAN = 40, SUB = 41, DIV = 42, SQUEEZE = 43, UNIDIRECTIONAL_SEQUENCE_LSTM = 44, STRIDED_SLICE = 45, BIDIRECTIONAL_SEQUENCE_RNN = 46, EXP = 47, TOPK_V2 = 48, SPLIT = 49, LOG_SOFTMAX = 50, // DELEGATE is a special op type for the operations which are delegated to // other backends. // WARNING: Experimental interface, subject to change DELEGATE = 51, BIDIRECTIONAL_SEQUENCE_LSTM = 52, CAST = 53, PRELU = 54, MAXIMUM = 55, ARG_MAX = 56, MINIMUM = 57, LESS = 58, NEG = 59, PADV2 = 60, GREATER = 61, GREATER_EQUAL = 62, LESS_EQUAL = 63, SELECT = 64, SLICE = 65, SIN = 66, TRANSPOSE_CONV = 67, SPARSE_TO_DENSE = 68, TILE = 69, EXPAND_DIMS = 70, EQUAL = 71, NOT_EQUAL = 72, LOG = 73, SUM = 74, SQRT = 75, RSQRT = 76, SHAPE = 77, POW = 78, ARG_MIN = 79, FAKE_QUANT = 80, REDUCE_PROD = 81, REDUCE_MAX = 82, PACK = 83, LOGICAL_OR = 84, ONE_HOT = 85, LOGICAL_AND = 86, LOGICAL_NOT = 87, UNPACK = 88, REDUCE_MIN = 89, FLOOR_DIV = 90, REDUCE_ANY = 91, SQUARE = 92, ZEROS_LIKE = 93, FILL = 94, FLOOR_MOD = 95, RANGE = 96, RESIZE_NEAREST_NEIGHBOR = 97, LEAKY_RELU = 98, SQUARED_DIFFERENCE = 99, MIRROR_PAD = 100, ABS = 101, SPLIT_V = 102, UNIQUE = 103, CEIL = 104, REVERSE_V2 = 105, ADD_N = 106, GATHER_ND = 107, COS = 108, WHERE = 109, RANK = 110, ELU = 111, REVERSE_SEQUENCE = 112, MATRIX_DIAG = 113, QUANTIZE = 114, MATRIX_SET_DIAG = 115, ROUND = 116, HARD_SWISH = 117, IF = 118, WHILE = 119, NON_MAX_SUPPRESSION_V4 = 120, NON_MAX_SUPPRESSION_V5 = 121, SCATTER_ND = 122, SELECT_V2 = 123, DENSIFY = 124, SEGMENT_SUM = 125, BATCH_MATMUL = 126, PLACEHOLDER_FOR_GREATER_OP_CODES = 127, CUMSUM = 128, CALL_ONCE = 129, BROADCAST_TO = 130, RFFT2D = 131, CONV_3D = 132, IMAG=133, REAL=134, COMPLEX_ABS=135, HASHTABLE = 136, HASHTABLE_FIND = 137, HASHTABLE_IMPORT = 138, HASHTABLE_SIZE = 139, REDUCE_ALL = 140, CONV_3D_TRANSPOSE = 141, VAR_HANDLE = 142, READ_VARIABLE = 143, ASSIGN_VARIABLE = 144, BROADCAST_ARGS = 145, RANDOM_STANDARD_NORMAL = 146, BUCKETIZE = 147, RANDOM_UNIFORM = 148, MULTINOMIAL = 149, GELU = 150, DYNAMIC_UPDATE_SLICE = 151, RELU_0_TO_1 = 152, UNSORTED_SEGMENT_PROD = 153, UNSORTED_SEGMENT_MAX = 154, UNSORTED_SEGMENT_SUM = 155, ATAN2 = 156, UNSORTED_SEGMENT_MIN = 157, SIGN = 158 }

fengyuentau

Thank you! 👍

alalek · 2023-02-10T22:44:29Z

Updated buildbot CI:

opencv-ubuntu:20.04 image enables flatbuffers (Linux OpenCL / Linux AVX2 builders from pre-commits).

alalek

LGTM 👍 Thank you!

dkurt · 2023-02-13T13:29:44Z

@alalek, @fengyuentau, thanks for the review!

TFLite models importer * initial commit * Refactor TFLiteImporter * Better FlatBuffers detection * Add permute before 4D->3D reshape * Track layers layout * TFLite Convolution2DTransposeBias layer * Skip TFLite tests without FlatBuffers * Fix check of FlatBuffers in tests. Add readNetFromTFLite from buffer * TFLite Max Unpooling test * Add skip for TFLite unpooling test * Revert DW convolution workaround * Fix ObjC bindings * Better errors handling * Regenerate TFLite schema using flatc * dnn(tflite): more checks, better logging * Checks for unimplemented fusion. Fix tests

dkurt mentioned this pull request Jan 24, 2023

Test data for TFLite importer opencv/opencv_extra#1042

Merged

dkurt commented Jan 24, 2023

View reviewed changes

modules/dnn/src/layers/fast_convolution/fast_convolution.cpp Outdated Show resolved Hide resolved

dkurt commented Jan 24, 2023

View reviewed changes

modules/dnn/test/test_tflite_importer.cpp Show resolved Hide resolved

This was referenced Jan 24, 2023

TFLite deep learning models #13918

Closed

Tensorflow Lite frontend openvinotoolkit/openvino#14977

Merged

dkurt changed the title ~~[WIP] TFLite models importer~~ TFLite models importer Jan 27, 2023

dkurt marked this pull request as ready for review January 27, 2023 14:27

dkurt added 11 commits January 30, 2023 10:55

initial commit

04e1c10

Refactor TFLiteImporter

522293a

Better FlatBuffers detection

08b809d

Add permute before 4D->3D reshape

c38f49f

Track layers layout

18c73bc

TFLite Convolution2DTransposeBias layer

e5e5949

Skip TFLite tests without FlatBuffers

9a92dc1

Fix check of FlatBuffers in tests. Add readNetFromTFLite from buffer

08307ea

TFLite Max Unpooling test

1f289a3

Add skip for TFLite unpooling test

a92d79d

Revert DW convolution workaround

b9efcdf

dkurt force-pushed the dnn_tflite branch from 9432fcf to b9efcdf Compare January 30, 2023 08:08

dkurt mentioned this pull request Jan 30, 2023

cv2.error: OpenCV(4.5.4-pre) /home/pi/opencv/modules/core/src/persistence.cpp:682: error: (-5:Bad argument) Input file is invalid in function 'open' #21516

Closed

dkurt commented Jan 31, 2023

View reviewed changes

cmake/OpenCVFindFlatBuffers.cmake Show resolved Hide resolved

dkurt commented Feb 1, 2023

View reviewed changes

modules/dnn/include/opencv2/dnn/dnn.hpp Show resolved Hide resolved

Fix ObjC bindings

1f7b617

dkurt requested review from fengyuentau and alalek February 6, 2023 11:17

alalek reviewed Feb 6, 2023

View reviewed changes

dkurt added 2 commits February 7, 2023 16:05

Better errors handling

9f4bea1

Regenerate TFLite schema using flatc

0d0a2d2

fengyuentau reviewed Feb 8, 2023

View reviewed changes

fengyuentau approved these changes Feb 9, 2023

View reviewed changes

alalek and others added 2 commits February 9, 2023 23:15

dnn(tflite): more checks, better logging

3d78c95

Checks for unimplemented fusion. Fix tests

2626453

alalek approved these changes Feb 13, 2023

View reviewed changes

alalek merged commit 76350cd into opencv:4.x Feb 13, 2023

This was referenced Feb 18, 2023

dnn(tflite): add 3rdparty flatbuffers with pre-generated schema #23274

Merged

dnn(tflite): fix wrong axis normalization #23287

Merged

Jonathhhan mentioned this pull request Mar 10, 2023

ofThreads Emscripten openframeworks/openFrameworks#7328

Merged

dkurt deleted the dnn_tflite branch March 14, 2023 07:05

asmorkalov mentioned this pull request May 31, 2023

(5.x) Merge 4.x #23718

Merged

Uh oh!

TFLite models importer #23161

TFLite models importer #23161

Uh oh!

Conversation

dkurt commented Jan 20, 2023 • edited by alalek Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

How to build (tested on Ubuntu 20.04)

How to build for OpenCV.js:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fengyuentau left a comment

Choose a reason for hiding this comment

Uh oh!

alalek commented Feb 10, 2023

Uh oh!

alalek left a comment

Choose a reason for hiding this comment

Uh oh!

dkurt commented Feb 13, 2023

Uh oh!

Uh oh!

dkurt commented Jan 20, 2023 •

edited by alalek

Loading