CARVIEW |
Navigation Menu
-
Notifications
You must be signed in to change notification settings - Fork 5.8k
[AutoParallel] FillZeroForEmpty* support AutoParallel #58716
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AutoParallel] FillZeroForEmpty* support AutoParallel #58716
Conversation
… grad_node_info_autoparallel_1
paddle/fluid/eager/grad_node_info.h
Outdated
@@ -178,6 +190,8 @@ class GradSlotMeta { | |||
// Save the dist attr of the forward input Tensor for proper resharding | |||
// operation when compute the input Tensor's gradient | |||
phi::distributed::TensorDistAttr dist_attr_; | |||
phi::DDim dist_tensor_global_dims_; | |||
bool is_dist_meta{false}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
属性需要加下划线吗?is_dist_meta_
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
好的好的,疏忽了
if (!IsRunAutoParallel()) {{ | ||
egr::EagerUtils::FillZeroForEmptyGradInput(&grads[{fwd_position}], input_metas[{fwd_position}]); | ||
}} | ||
egr::EagerUtils::FillZeroForEmptyGradInput(&grads[{fwd_position}], input_metas[{fwd_position}]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
欢哥,FillZeroForEmpty*
会创建一个全0的Grad Tensor占有显存,但pp的非计算op前反向都不能占显存,需要控制一下进不进这个函数:
bool rank_is_in_current_mesh = true;
if (IsRunAutoParallel()) {
auto mesh = std::static_pointer_cast<phi::distributed::DistTensor>(grads[{fwd_position}].impl())->dist_attr().process_mesh();
rank_is_in_current_mesh = phi::distributed::IsCurRankInMesh(mesh);
}
if (rank_is_in_current_mesh) {{
egr::EagerUtils::FillZeroForEmptyGradInput(&grads[{fwd_position}], input_metas[{fwd_position}]);
}}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
哦,这个是通过下面这种逻辑来控制的
if (tensor_meta.dims.size() != -1) {
auto tensor_with_zero =
paddle::experimental::full(phi::vectorize(tensor_meta.dims),
0.0,
tensor_meta.dtype,
grad_in_meta.GetPlace());
*(static_castphi::distributed::DistTensor*(in_grad->impl().get())
->unsafe_mutable_value()) =
*(static_castphi::DenseTensor*(tensor_with_zero.impl().get()));
}
…into grad_node_info_autoparallel_1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
…8716) * grad_node_info.cc support autoparallel 1
…8716) * grad_node_info.cc support autoparallel 1
PR types
Others
PR changes
Others
Description
FillZeroForEmpty* support AutoParallel
8卡本地测试OK
Pcard-73145