你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@Xreki,

This PR should resolve the problem of NCCL not being able to overlap with compute kernels if tensor parallelism is enabled.

It was mentioned that once the CUDA_DEVICE_MAX_CONNECTIONS flag is set, overlapping operations become entirely unfeasible. This limitation arises from the blocking introduced by the use of cudaEventDefault, since there's an inherent conflict between CUDA_DEVICE_MAX_CONNECTIONS and cudaEventDefault. We can configure the event to utilize cudaEventDisableTiming, so we could use CUDA_DEVICE_MAX_CONNECTIONS.

Origin Timeline（use paddle.ones to make all_reduce(dx) occupying SM before matmul(dw)）

Timeline with fix PR（remove paddle.ones, and export CUDA_DEVICE_MAX_CONNECTIONS=1）

It fixes the problem of mp_async_allreduce overlap failure.

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ eee4017
❌ gongweibao

@eee4017 ,

PR-CI-ROCM-Compile:

2023-09-23 00:18:56 FAILED: paddle/fluid/platform/CMakeFiles/device_event_test.dir/device_event_test.cc.o
...
2023-09-23 00:18:56 ../paddle/fluid/platform/device_event_test.cc: In member function 'virtual void DeviceEvent_CUDA_Test::TestBody()':
2023-09-23 00:18:56 ../paddle/fluid/platform/device_event_test.cc:89:26: error: no matching function for call to 'paddle::platform::DeviceEvent::DeviceEvent(phi::GPUPlace&)'
2023-09-23 00:18:56    DeviceEvent event(place);

This branch was cherry-picked by #57656. Close this PR.

Hi @eee4017 , #57656 是将该PR的修改cherry-pick到incubate/new_frl分支，develop分支仍需求合入该PR。