[Fix] Reduce unnecessary GpuMemGetInfo in `Tensor.to` #70359

HydrogenSulfate · 2024-12-20T02:36:01Z

PR Category

Performance Optimization

PR Types

Performance

Description

Pcard-75624

问题

Tensor.to方法中，对于device的处理是当src.place在GPU上时会通过core.gpu_memory_available()计算当前设备上的剩余可用显存，接着后续一系列操作。这个过程中core.gpu_memory_available()会调用cudaMemGetInfo，而当.to操作一多时，会非常频繁地调用cudaMemGetInfo，导致拖慢整体动态图的运行速度（cudaMemGetInfo看起来会产生一个CPU阻塞效果，API文档也没有相关说明）。

Paddle/paddle/phi/core/platform/device/gpu/gpu_info.cc

Lines 71 to 78 in 5859a05

    
           void GpuMemoryUsage(size_t *available, size_t *total) { 
        
             size_t actual_available, actual_total; 
        
             RecordedGpuMemGetInfo(available, 
        
                                   total, 
        
                                   &actual_available, 
        
                                   &actual_total, 
        
                                   platform::GetCurrentDeviceId()); 
        
           }

解决方案

修改进入core.gpu_memory_available分支的判断条件，仅当t.place(source place)与device(dst place)设备类型不同，或者同设备类型但卡号不同时，才进入该copy分支，而不是t.place只要在GPU上就直接进入分支。从VLOG来看，这一修改可以避免所有step中的cudaMemGetInfo操作，耗时恢复正常。

nsys可以看到每个step内已经没有cudaMemGetInfo

paddle-bot · 2024-12-20T02:36:05Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

reduce unnecessary GpuMemGetInfo call by strict cuda copy condition

96bd020

HydrogenSulfate changed the title ~~reduce unnecessary GpuMemGetInfo call by strict cuda copy condition~~ [Fix] Reduce unnecessary GpuMemGetInfo in Tensor.to Dec 20, 2024

zyfncg approved these changes Dec 20, 2024

View reviewed changes

HydrogenSulfate merged commit 9e991d1 into PaddlePaddle:develop Dec 20, 2024
27 of 28 checks passed

HydrogenSulfate deleted the reduce_unnecessary_GpuMemGetInfo branch December 20, 2024 10:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Fix] Reduce unnecessary GpuMemGetInfo in `Tensor.to` #70359

[Fix] Reduce unnecessary GpuMemGetInfo in `Tensor.to` #70359

Uh oh!

HydrogenSulfate commented Dec 20, 2024 •

edited

Loading

Uh oh!

paddle-bot bot commented Dec 20, 2024

Uh oh!

Uh oh!

Uh oh!

	void GpuMemoryUsage(size_t available, size_t total) {
	size_t actual_available, actual_total;
	RecordedGpuMemGetInfo(available,
	total,
	&actual_available,
	&actual_total,
	platform::GetCurrentDeviceId());
	}

[Fix] Reduce unnecessary GpuMemGetInfo in Tensor.to #70359

[Fix] Reduce unnecessary GpuMemGetInfo in Tensor.to #70359

Uh oh!

Conversation

HydrogenSulfate commented Dec 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Category

PR Types

Description

问题

解决方案

Uh oh!

paddle-bot bot commented Dec 20, 2024

Uh oh!

Uh oh!

Uh oh!

[Fix] Reduce unnecessary GpuMemGetInfo in `Tensor.to` #70359

[Fix] Reduce unnecessary GpuMemGetInfo in `Tensor.to` #70359

HydrogenSulfate commented Dec 20, 2024 •

edited

Loading