| CARVIEW |
Navigation Menu
-
Notifications
You must be signed in to change notification settings - Fork 5.8k
[SOT] Add kernel stats in SOT mode #67560
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SOT] Add kernel stats in SOT mode #67560
Conversation
SigureMo
requested review from
XiaoguangHu01,
zhiqiu,
Xreki,
qili93 and
Aurelius84
as code owners
|
你的PR提交成功，感谢你对开源项目的贡献! |
paddle/fluid/framework/new_executor/instruction/phi_kernel_instruction.cc
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
你要不要加一个单测？
| @@ -20,52 +20,64 @@ limitations under the License. */ | |||
|
|
|||
| namespace phi { | |||
|
|
|||
| #define TRACER_EVENT_TYPES \ | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| #define TRACER_EVENT_TYPES \ | |
| #define TRACER_EVENT_TYPES(X) \ |
一般我们推荐把宏X也传入进来。否则必须要求外部定义的处理名字必须是X。在框架中更多的是用下划线 _ 来表示内层宏明
|
|
||
| # TODO(SigureMo): Split into multiple files by visitor type | ||
| class KernelStatsVisitor(EventVisitor): | ||
| SKIP_KERNEL_NAMES = ["full", "full_int_array", "shadow_feed"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这里可以使用set？下面大多数是in判断，set更高效一些？
| self.started = False | ||
| self.record_event = None | ||
|
|
||
| def _kernel_stats(self, prof): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| def _kernel_stats(self, prof): | |
| def _kernel_stats(self, prof) -> str: |
这里要加typehint 么？
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
均已修改
你要不要加一个单测？
加了个烟雾测试
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work!
| @@ -135,6 +135,7 @@ PhiKernelInstruction::PhiKernelInstruction( | |||
| .data(); | |||
| auto kernel_result = phi::KernelFactory::Instance().SelectKernelOrThrowError( | |||
| kernel_name, kernel_key); | |||
| kernel_name_ = kernel_name; | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
有一些算子是不是会走legacy_instruction?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
看了下调用路径，这些 legacy_instruction 基本都是通信算子，现阶段可以先不考虑～
Paddle/paddle/fluid/pir/dialect/operator/utils/utils.cc
Lines 38 to 72 in 30625fd
| const std::unordered_set<std::string> LegacyOpList = { | |
| LoadCombineOp::name(), | |
| CConcatOp::name(), | |
| CBroadcast_Op::name(), | |
| CSyncCommStream_Op::name(), | |
| DistributedPushSparseOp::name(), | |
| SendV2Op::name(), | |
| RecvV2Op::name(), | |
| CAllreduceProd_Op::name(), | |
| CAllreduceSumOp::name(), | |
| CAllreduceSum_Op::name(), | |
| CAllreduceAvgOp::name(), | |
| CAllreduceAvg_Op::name(), | |
| CReduceSumOp::name(), | |
| CReduceSum_Op::name(), | |
| CAllreduceMax_Op::name(), | |
| CAllreduceMaxOp::name(), | |
| CAllreduceMin_Op::name(), | |
| CAllgatherOp::name(), | |
| CSoftmaxWithCrossEntropyOp::name(), | |
| CSoftmaxWithCrossEntropyGradOp::name(), | |
| CSplitOp::name(), | |
| PushDenseOp::name(), | |
| SoftReluOp::name(), | |
| SoftReluGradOp::name(), | |
| CReduceAvgOp::name(), | |
| CReduceAvg_Op::name(), | |
| CReduceMaxOp::name(), | |
| CReduceMinOp::name(), | |
| CReduceProdOp::name(), | |
| CScatterOp::name(), | |
| PullBoxSparseOp::name(), | |
| PushBoxSparseOp::name(), | |
| PushSparseV2Op::name(), | |
| SendAndRecvOp::name()}; |
|
|
||
|
|
||
| class SotStepProfilerGuard: | ||
| EXPORT_CHROME_TRACING_PATH = "./sot-chrome-tracing/" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这个导出的所属目录是哪个？
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
就是当前目录下的 sot-chrome-tracing，当前这个只是内部调试用的，只能修改代码启用，还没有暴露通过 FLAG 开启的方式
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
PR Category
Execute Infrastructure
PR Types
Devs
Description
为 SOT 添加 kernel 信息统计工具，基于 Paddle Profiler，添加两个
TracerEventType，分别为DygraphKernelCall和StaticKernelCall，用于标记 kernel 调用，这样通过 profiler 就可以直接获取动态图调用的 Kernel 数、时间以及静态图调用的 Kernel 数和时间，以用于 SOT 指标建设，相比于之前的 OP 数量，这会是一个更加稳定的绝对参考值使用方式：
Note
请不要和
Profiler混用，本工具本质是会额外创建一个Profiler实例，相互之间会有影响TODOs:
需要确认下基本都是通信算子，暂时不统计Future works
Pcard-67164