CARVIEW |
Navigation Menu
-
Notifications
You must be signed in to change notification settings - Fork 5.8k
[Auto Parallel] Add spmd rule No.12 for label_smooth and label_smooth ops. #72845
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Auto Parallel] Add spmd rule No.12 for label_smooth and label_smooth ops. #72845
Conversation
你的PR提交成功,感谢你对开源项目的贡献! |
… add_spmd_label_smooth
label_smooth
,expand_as
) ops and their grad ops} | ||
SpmdInfo unary_spmd_info = ElementwiseUnaryInferSpmd(label); | ||
unary_spmd_info.first.push_back(TensorDistAttr()); | ||
VLOG(4) << "LabelSmoothInferSpmd:"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it is better to put this line before L30/L32 and make the logs more understandable? e.g.
before L30 VLOG(4) << "LabelSmoothInferSpmd call ElementwiseBinaryInferSpmd:"
before L32 VLOG(4) << "LabelSmoothInferSpmd call ElementwiseUnaryInferSpmd:"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!done
self.assertEqual(inferred_input_dist_attrs[1].dims_mapping, [-1, -1]) | ||
self.assertEqual(inferred_output_dist_attrs[0].dims_mapping, [0, 1, -1]) | ||
|
||
# shape: [16, 16, 16], [1, 16]. [0, 1, -1], [-1, 0] --> [0, 1, -1], [-1, -1], [0, 1, -1] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we need more unit test case to check:
if # shape: [16, 16, 16], [1, 16]. [0, -1, 1], [-1, 1]
, what result should be
if # shape: [16, 16, 16], [1, 16]. [0, -1, 1], [-1, 0]
, what result should be
if # shape: [16, 16, 16], [1, 16]. [0, -1, 1], [-1, -1]
, what result should be
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the second station # shape: [16, 16, 16], [1, 16]. [0, -1, 1], [-1, 0]
Because of reuse the ElementwiseBinaryInferSpmd
, now it will cause error now when meet 1 and 0 with same axes (due to the ShardingMergeForAxis
).
The following error is below, and I check the error now rather than implement it
(Unimplemented) Tensor Axis[c] is Sharded by two different mesh dimension [0] and [1]. (at ../paddle/phi/infermeta/spmd_rules/utils.cc:70)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
According to the calculation logic of this operator, if # shape: [16, 16, 16], [1, 16]
and dims_mapping: [0, -1, 1], [-1, 0]
the result of dims_mapping should be [0, -1, 1], [-1, 1], [0, -1, 1]
? if so, we should add some necessary checks and modifications to dims_mapping of prior_dist before calling ElementwiseBinaryInferSpmd
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. If # shape: [16, 16, 16], [1, 16]
and dims_mapping: [0, -1, 1], [-1, 0]
the result of dims_mapping should be [0, -1, 1], [-1, 1], [0, -1, 1]
If label's rank is greater than prior_dist's axis. It has an extra_dims_mapping. When prior_dist's dims_mappin in it, replicate it to -1
first. It will work better than before.
(In addition, I think in the future ElementwiseBinaryInferSpmd
also should deal with this station
… add_spmd_label_smooth
/re-run all-failed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
check_dim_mapping(forward_info.first[0], {0, 1, -1}); | ||
const phi::distributed::ArgDistAttr& attr = forward_info.first[1]; | ||
if (paddle::holds_alternative<phi::distributed::TensorDistAttr>(attr)) { | ||
EXPECT_EQ(paddle::get<phi::distributed::TensorDistAttr>(attr), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.

There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
好的好的
… ops. (PaddlePaddle#72845) * add spmd_rule for label_smooth * update * fix bugs * apply review * revert due to PaddlePaddle#72753 * revert change for expand_as * Add more tests * apply review * refine code
PR Category
Auto Parallel
PR Types
New features
Description
查看 expand_as 当前已有推导规则实现,故仅仅在 yaml 中进行注册,此外修复了 target_shape 的类型不匹配问题