[Inference]Support TensorRT execute in PIR #64995

YuanRisheng · 2024-06-07T09:21:25Z

PR Category

Inference

PR Types

New features

Description

Pcard-71500
本PR主要做如下工作：
1，开发TensorRTEngineOp，实现TensorRT在PIR下核心表达机制和注册注册机制
2，实现TensorRT在PIR下核心执行机制（lower to kernel变换，TRTEngineInstruction）
3，完成TensorRT三方基础封装组件重新开发优化
4，Python端支持使用tensorrt构建PIR图结构功能
5，完成CollectShape模块开发，支持运行时存储shape信息并且对shape范围进行统计，根据Program Value获取对应shape range
6，TensorRT在PIR下基础功能单测开发

TODO:
支持保存trt engine

paddle-bot · 2024-06-07T09:21:30Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

vivienfanghuagood · 2024-06-14T02:50:26Z

paddle/fluid/framework/new_executor/instruction/tensorrt_engine_instruction.h

+#ifdef PADDLE_WITH_TENSORRT
+#include "paddle/fluid/framework/new_executor/instruction/instruction_base.h"
+#include "paddle/fluid/framework/new_executor/pir_adaptor/pir_adaptor_util.h"
+#include "paddle/fluid/inference/tensorrt/engine.h"


这个engine类建议抽出来吧。

engine.h里TensorRTEngine类本身就是独立的，没理解这个类抽出来是指的什么

vivienfanghuagood · 2024-06-14T02:51:58Z

paddle/fluid/framework/new_executor/instruction/tensorrt_engine_instruction.cc

+                        max_input_shape_str));
+}
+
+static phi::DataType TRT2FluidDataType(nvinfer1::DataType type) {


TRT2PaddleDataType

谢谢，我改一下

vivienfanghuagood · 2024-06-14T02:55:57Z

paddle/fluid/framework/new_executor/instruction/tensorrt_engine_instruction.cc

+  auto op_attributes = op->attributes();
+  trt_engine_ = static_cast<TensorRTEngine *>(
+      op_attributes.at("engine").dyn_cast<pir::PointerAttribute>().data());
+  max_batch_size_ =


max_batch_size_这个模式可以不要了

谢谢，我改一下

vivienfanghuagood · 2024-06-14T02:56:19Z

paddle/fluid/framework/new_executor/CMakeLists.txt

@@ -24,7 +29,8 @@ set(standalone_executor_deps
    garbage_collector
    executor_gc_helper
    device_event_base
-    framework_proto)
+    framework_proto


这个必须依赖framework吗？

这里tensorrt_engine_instruction与其他instruction共同编入standalone_executor这个target中，这个target对framework_proto是有依赖的，单独的tensorrt_engine_instruction不依赖

yuanlehome · 2024-06-17T03:01:56Z

paddle/fluid/framework/new_executor/CMakeLists.txt

@@ -24,7 +29,8 @@ set(standalone_executor_deps
    garbage_collector
    executor_gc_helper
    device_event_base
-    framework_proto)
+    framework_proto
+    pir_transforms)


这里似乎存在循环依赖，pir_transforms中的constant_folding_pass是依赖standalone_executor的

这里我先还原吧，不过需要注意这里的问题：本身就是循环依赖，standalone_executor也要用到pd_op_to_kernel_pass, 这个是适配tensorrt前就有的问题，因为我在开发代码过程中偶现了这个问题就加了一下，后续如果不解决代码本身的循环依赖，依然可能暴露符号找不到的问题

yuanlehome · 2024-06-17T03:12:11Z

paddle/fluid/framework/new_executor/pir_interpreter.cc

+#ifdef PADDLE_WITH_TENSORRT
+    } else if (op.dialect()->name() == "trt_op") {
+      CREATE_INSTR(TensorRTEngineInstruction);
+#endif


这里为啥不是trt_kernel，其他判断都是xxx_kernel，这里突然是个xxx_op，挺奇怪的

这里是故意这样做的，因为这里是lower到kernel后的操作，但是对于tensorrt适配来说，Lower前和lower后的ir表示基本没啥变化，为了减少重复代码，这里没有再单独开发tensorrt kernel

yuanlehome · 2024-06-17T03:14:52Z

paddle/fluid/pir/dialect/operator/ir/op_dialect.h

+  void PrintType(pir::Type type,
+                 std::ostream& os) const override;  // 用于打印type有关信息
+  void PrintAttribute(pir::Attribute type, std::ostream& os)
+      const override;  // 用于打印attribute有关信息
+
+  pir::OpPrintFn PrintOperation(
+      pir::Operation* op) const override;  // 用于打印operation有关信息


中文注释是否可以改成英文注释？

谢谢，我改一下

yuanlehome · 2024-06-17T03:16:01Z

paddle/fluid/pir/dialect/operator/ir/tensorrt_op.cc

+const char* TensorRTEngineOp::attributes_name[8] = {"engine",
+                                                    "max_batch_size",
+                                                    "workspace_size",
+                                                    "allow_build_at_runtime",
+                                                    "input_names",
+                                                    "output_names",
+                                                    "origin_output_rank",
+                                                    "origin_outputs_dtype"};


PIR下的TensorRTEngineOp一共有哪些属性需要再讨论下

是的，目前这里只是必要的属性先加入了，这个是迭代完善的过程，后续代码还会逐渐完善

yuanlehome · 2024-06-17T03:17:36Z

paddle/fluid/pir/dialect/operator/ir/tensorrt_op.h

+    : public pir::Op<TensorRTEngineOp, paddle::dialect::OpYamlInfoInterface> {
+ public:
+  using Op::Op;
+  static const char *name() { return "trt_op.tensorrt_engine_op"; }


总感觉，不需要一个额外的trt_op dialect，tensorrt_engine_op仍属于pd_op下比较合适？

pd_op下的op都有共性，比如从yaml里提取的op信息，带有Interface等等，tensorrt_engine_op不具备pd_op下op的共性，故单独抽出来更好

目前已删除trt_op dialect，避免过度冗余设计，后续有需求再加上

yuanlehome · 2024-06-17T03:19:15Z

paddle/fluid/pir/transforms/pd_op_to_kernel_pass.cc

+    if (paddle::dialect::IsTensorRTOp(op_item)) {
+      HandleForTensorRTOp(ctx,
+                          op_item,
+                          kernel_key,
+                          place,
+                          map_op_pair,
+                          map_value_pair,
+                          new_block);
+      continue;
+    }


PIR下所有TRT相关代码，建议都使用PADDLE_WITH_TENSORRT宏包裹下

谢谢，我改一下

vivienfanghuagood · 2024-07-01T08:54:23Z

paddle/fluid/framework/new_executor/collect_shape_manager.cc

+              if (shape[d] < min_shape[d]) min_shape[d] = shape[d];
+              if (shape[d] > max_shape[d]) max_shape[d] = shape[d];
+            }
+            opt_shape[d] = ShapeMaxFreq(counter);


挺好的设计哈哈

vivienfanghuagood · 2024-07-01T09:12:29Z

paddle/fluid/pybind/pybind.cc

+           paddle::framework::ShapeMode shape_mode) -> py::list {
+          py::list res;
+          paddle::framework::CollectShapeManager::Instance()
+              .StatisticShapeRangeInfo();


我看到这个通过局部对象控制是否shape是否ready，从而是否收集shape。我觉得是否把StatisticShapeRangeInfo也单独暴露一个方法出来，这样首次collect shape之后，后续还可以继续collect shape。

不需要单独暴露出来，这里对局部变量进行了重置来支持重新collect shape

yuanlehome · 2024-07-02T07:22:07Z

paddle/fluid/pir/dialect/operator/ir/tensorrt_op.h

+    : public pir::Op<TensorRTEngineOp, paddle::dialect::OpYamlInfoInterface> {
+ public:
+  using Op::Op;
+  static const char *name() { return "pd_op.tensorrt_engine_op"; }


旧IR下算子名字叫tensorrt_engine，这里建议沿用此名，无需_op后缀？

谢谢，我修改一下

risemeup1

LGTM

risemeup1

LGTM

XieYunshen

LGTM

XiaoguangHu01

LGTM

This reverts commit 101bf6e.

* adapt tensorrt * fix compile bugs * delete thirdparty * add unittest * fix py3 compile * fix kunlun200 * fix windows inference * fix windows bug * polish code * polish code * polish code * support build trt_op in python * rename construction params * fix bug * fix compile bugs * support collect shape * support re-collect shape * rename tensorrt op * polish code * add debug attr * delete member in tensorrt engine instruction * remove mutable_data * fix compile

adapt tensorrt

a42b7f2

YuanRisheng added 8 commits June 7, 2024 09:42

fix compile bugs

f84d629

resolve conflict

4a861c4

delete thirdparty

b5292e1

add unittest

3eb6609

fix py3 compile

51ff0c4

fix kunlun200

210bd6b

fix windows inference

9791a2d

fix windows bug

83d9b67

vivienfanghuagood reviewed Jun 14, 2024

View reviewed changes

polish code

622522b

yuanlehome reviewed Jun 17, 2024

View reviewed changes

YuanRisheng added 5 commits June 17, 2024 06:11

polish code

6a992fa

polish code

b8ef67e

support build trt_op in python

b128f5a

rename construction params

ad2c0c9

fix bug

a5f3c94

PaddlePaddle locked and limited conversation to collaborators Jun 25, 2024

PaddlePaddle unlocked this conversation Jun 25, 2024

YuanRisheng added 3 commits June 25, 2024 10:54

fix compile bugs

14b5801

support collect shape

885e87b

resolve conflicgt

1570220

vivienfanghuagood reviewed Jul 1, 2024

View reviewed changes

support re-collect shape

2095849

yuanlehome reviewed Jul 2, 2024

View reviewed changes

YuanRisheng added 3 commits July 2, 2024 08:13

rename tensorrt op

b9b1769

polish code

aecfa26

add debug attr

56e34d6

delete member in tensorrt engine instruction

df06ed0

vivienfanghuagood approved these changes Jul 12, 2024

View reviewed changes

risemeup1 previously approved these changes Jul 12, 2024

View reviewed changes

remove mutable_data

a85f41d

YuanRisheng dismissed risemeup1’s stale review via a85f41d July 12, 2024 08:22

risemeup1 previously approved these changes Jul 12, 2024

View reviewed changes

fix compile

27b0cf1

YuanRisheng dismissed risemeup1’s stale review via 27b0cf1 July 12, 2024 09:53

risemeup1 approved these changes Jul 15, 2024

View reviewed changes

luotao1 approved these changes Jul 15, 2024

View reviewed changes

zhangbo9674 approved these changes Jul 15, 2024

View reviewed changes

XieYunshen approved these changes Jul 15, 2024

View reviewed changes

XiaoguangHu01 approved these changes Jul 15, 2024

View reviewed changes

YuanRisheng merged commit 101bf6e into PaddlePaddle:develop Jul 15, 2024
30 of 32 checks passed

risemeup1 added a commit that referenced this pull request Jul 15, 2024

Revert "[Inference]Support TensorRT execute in PIR (#64995)"

42d8c2a

This reverts commit 101bf6e.

risemeup1 mentioned this pull request Jul 15, 2024

Revert "[Inference]Support TensorRT execute in PIR" #66067

Closed

lizexu123 mentioned this pull request Aug 7, 2024

[Paddle TensorRT] Refactor subgrah segmentation and operator convter of Paddle TensorRT base on PIR. #67054

Merged

[Inference]Support TensorRT execute in PIR #64995

[Inference]Support TensorRT execute in PIR #64995

Uh oh!

Conversation

YuanRisheng commented Jun 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Category

PR Types

Description

Uh oh!

paddle-bot bot commented Jun 7, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

risemeup1 left a comment

Choose a reason for hiding this comment

Uh oh!

risemeup1 left a comment

Choose a reason for hiding this comment

Uh oh!

XieYunshen left a comment

Choose a reason for hiding this comment

Uh oh!

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

YuanRisheng commented Jun 7, 2024 •

edited

Loading