| CARVIEW |
Navigation Menu
-
Notifications
You must be signed in to change notification settings - Fork 5.8k
[Auto Parallel] support global mesh output with pipeline #69628
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
你的PR提交成功，感谢你对开源项目的贡献! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
在单测中加一下这个case吧
| if "pp" in g_mesh.dim_names: | ||
| g_mesh = g_mesh.get_mesh_with_dim("pp") | ||
| else: | ||
| return |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
为什么output就不需要这个return？
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
output插入的是shard_tensor，这个是必须的，有没有pp都需要标记一下。但是input插入的reshard，非pp的情况下，大家mesh相同，所以就不用插reshard了
|
|
||
| def forward_pre_hook(layer, input): | ||
| new_input = [] | ||
| for t in input: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
input不需要考虑单个input的场景么？
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
经过测试，单个Input也是一个tuple
deepllz
changed the title
| pp_config.get('split_spec'), | ||
| pp_config.get("global_output_layers", None), | ||
| pp_config.get("layers_with_global_input", None), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
统一只传config吧，然后在里面解析。如果以后还需要加东西的话，这个函数签名会变得无比的长。dp的我在弄了。
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
好的，在下一个PR中处理
| model, | ||
| optimizer, | ||
| split_spec, | ||
| global_output_layers=None, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
use global_spec to better understand
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
| optimizer, | ||
| split_spec, | ||
| global_output_layers=None, | ||
| layers_with_global_input=None, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
temporarily delete this parameter
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
PR Category
Auto Parallel
PR Types
Others
Description
Pcard-73145
中层API添加全局算子的并行方式