支持非均衡VPP编排的灵活模型层分配策略 #70230

zty-king · 2024-12-14T19:57:09Z

PR Category

Auto Parallel

PR Types

Others

Description

当前自动并行下非均衡vpp编排无法支持用户灵活进行模型层分配，即在任意设备上分配任意模型层数，当前仅支持，在每个设备上每个chunk中放入相同的模型层数。
以下举一个例子验证当前在任意设备分配任意模型层数存在的问题：
这里选择hidden_layer=8，layer_to_mesh=[mesh0,mesh0,mesh0,mesh0,mesh0,mesh1,mesh1,mesh1]（即0设备5层，1设备3层）
，运行发现，vpp编排能正常进行，但是编排方式是假灵活分配，如下图所示：

仍然把模型层数均匀分配到了每一个设备上，因此本项工作对上述内容进行了优化，优化效果如下图所示：

可以看到此时在第0，1，2层hidden_layer分配在0号设备，第3，4层分配在1号设备，第5，6层分配在0号设备，第7层分配在1号设备，与用户的layer_to_mesh（即0设备5层，1设备3层），保持一致。
方法简述：
1. 首先过滤掉op.dist_attr为None的ops，因为在后续调用get_pp_stage_by_process_mesh函数获取当前模型层的pp_stage时需要用到op.dist_attr.process_mesh。
2. 在同一个layer上的op对应的pp_stage是相同的，因此用struct_name区分不同层的op，并且每层只需要利用一个op来计算对应的pp_stage。
3. 确保每个设备的分配的layer数大于等于chunk数，即保证每个chunk中至少包含一个layer。
4. 设备内使用Round-Robin算法，对每个设备来说：设备中每个块轮循增加layer数，直到达到当前设备的指定数
5. 最终得到按用户意图分配来编排的结果

paddle-bot · 2024-12-14T19:57:14Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

From00

需要补充说明一下实现方案

From00 · 2024-12-16T08:51:28Z

python/paddle/distributed/auto_parallel/static/pir_pass.py

    # Step2: analysis whether the pp_stage is non-decreasing among segments
    # 1. if non_use_custom_mesh is True, the ops' process_mesh will be changed by vpp strategy
    # 2. if non_use_custom_mesh is False, the ops's process_mesh will not be changed.
    non_use_custom_mesh = _analyze_use_custom_mesh(ops, seg_method, pp_degree)

    # Step3: Get op index boundary, pp_stage, chunk_id, struct_names of each segment
+    if len(seg_struct_names) % num_chunks == 0:


用户标记不同stage不同layer数，是否也可能出现len(seg_struct_names) % num_chunks == 0的情况？

已经修改，适配所有情况

From00 · 2024-12-16T08:52:56Z

python/paddle/distributed/auto_parallel/static/pir_pass.py

+def get_user_layer_to_mesh(ops, seg_method, pp_degree, segment_nums):
+    pp_stage_list = []
+    for op in ops:
+        if _extract_seg_method(op, seg_method) and "pd_op" in op.name():


这里为什么要限制"pd_op" in op.name()？

有一部分op没有dist_attr（就是skip_op_list里面的，所以要过滤一下），但是计算op都是pd_op开头的，所以就用这个来做过滤了，参考的这里的代码：

From00 · 2024-12-16T08:56:13Z

python/paddle/distributed/auto_parallel/static/pir_pass.py

+            pp_stage_list.append(
+                get_pp_stage_by_process_mesh(op_mesh, pp_degree)
+            )
+    per_segment_op_nums = len(pp_stage_list) // segment_nums


如何理解这条公式？per_segment_op_nums表示每个layer中的op数量？

对的，每层layer中的op数

From00 · 2024-12-16T08:58:29Z

python/paddle/distributed/auto_parallel/static/pir_pass.py

+
+    user_layer_to_mesh = [
+        sum(pp_stage_list[i : i + per_segment_op_nums])
+        // len(pp_stage_list[i : i + per_segment_op_nums])


len(pp_stage_list[i : i + per_segment_op_nums])是不是等价于per_segment_op_nums？

这里修改成了更好理解的方式，每隔per_segment_op_nums取一次ID，即每层对应的process_id

From00 · 2024-12-20T06:44:53Z

python/paddle/distributed/auto_parallel/static/pir_pass.py

+    user_layer_to_stage_id = (
+        []
+    )  # User intent - a list corresponding to the model layer and the device. The key of the list is the number of the layer, and the value is the corresponding pp_stage.


Suggested change

user_layer_to_stage_id = (

[]

) # User intent - a list corresponding to the model layer and the device. The key of the list is the number of the layer, and the value is the corresponding pp_stage.

stage_ids=[]

stage_ids[i]表示第i层分配的stage编号，这样是否更简洁一些？

From00 · 2024-12-24T02:55:22Z

python/paddle/distributed/auto_parallel/static/pir_pass.py

    for idx, op in enumerate(ops):
        if len(seg_parts) == len(seg_struct_names):
            break
        struct_name = _extract_seg_method(op, seg_method)
+        if (
+            "pd_op" in op.name() and last_struct_name != struct_name


if op.name() not in skip_op_list
可读性更强些

From00 · 2024-12-24T02:59:57Z

python/paddle/distributed/auto_parallel/static/pir_pass.py

+            "pd_op" in op.name() and last_struct_name != struct_name
+        ):  # When traversing the ops, filter out the ops that need to be skipped. At the same time, according to the struct_name, ensure that the pp_stage of each layer is only recorded once.
+            last_struct_name = struct_name
+            user_layer_to_stage_id.extend(op.dist_attr.process_mesh.process_ids)


这里为何不是一个自增的stage_id?

From00 · 2024-12-24T03:02:01Z

python/paddle/distributed/auto_parallel/static/pir_pass.py

+        pp_stage_layer_num[i] = pp_stage_layer_num[i] + 1
+    assert all(
+        value >= vpp_degree for value in pp_stage_layer_num
+    ), "Make sure each segment is not empty"


这个提示信息和拦截条件不匹配

From00 · 2024-12-24T03:02:59Z

python/paddle/distributed/auto_parallel/static/pir_pass.py

+        pp_stage_layer_nums = pp_stage_layer_num[pp_stage]
+        for i in range(
+            0, pp_stage_layer_nums
+        ):  # The pp_stage uses a Round robin scheduling algorithm to allocate layers one by one.


较长的注释直接单独成一行

paddle-ci-bot · 2024-12-28T03:13:43Z

Sorry to inform you that 4f7e586's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

… pipeline_vpp

From00 · 2025-01-09T02:59:37Z

python/paddle/distributed/auto_parallel/static/pir_pass.py

    for idx, op in enumerate(ops):
        if len(seg_parts) == len(seg_struct_names):
            break
        struct_name = _extract_seg_method(op, seg_method)
+        if op.dist_attr is not None and last_struct_name != struct_name:
+            # When traversing the operations, filter out those without any ops where `has_attr` is `None`. At the same time, ensure that the `pp_stage` of each layer is recorded only once according to the `struct_name`.


这条注释只是把代码直白地翻译一遍，没有必要，注释应该是写无法直观从代码中获取到的信息。

From00 · 2025-01-09T03:00:11Z

python/paddle/distributed/auto_parallel/static/pir_pass.py

+        if op.dist_attr is not None and last_struct_name != struct_name:
+            # When traversing the operations, filter out those without any ops where `has_attr` is `None`. At the same time, ensure that the `pp_stage` of each layer is recorded only once according to the `struct_name`.
+            if (
+                get_pp_stage_by_process_mesh(


这个写法get_pp_stage_by_process_mesh的逻辑会被重复调用两遍

From00 · 2025-01-09T03:06:12Z

python/paddle/distributed/auto_parallel/static/pir_pass.py

+    for pp_stage in range(
+        0, pp_degree
+    ):  # Each pp_stage is assigned a number of tiers based on user intent.
+        pp_stage_layer_nums = pp_stage_layer_num[pp_stage]


Suggested change

pp_stage_layer_nums = pp_stage_layer_num[pp_stage]

pp_stage_layer_num = pp_stage_layer_nums[pp_stage]

变量命名要符合语法逻辑，比如加s表示复数

From00 · 2025-01-09T03:08:20Z

python/paddle/distributed/auto_parallel/static/pir_pass.py

+    assert all(
+        value >= vpp_degree for value in pp_stage_layer_num
+    ), "The number of layers on each pp_stage must not be less than the vpp_degree in the pp_stage to ensure that each chunk contains at least one layer."
+    seg_layer_num = [0] * num_chunks


Suggested change

seg_layer_num = [0] * num_chunks

# Each chunk is assigned a number of layers based on user intent.

seg_layer_num = [0] * num_chunks

用户在使用vpp的时候一般不会深入到chunk部分，所以这里不会按照用户意图去分配chunk中的layer数，而是按照用户意图分配每个pp_stage中的layer数，chunk中的layer数由底层代码在每个pp_stage中，按照Round-Robin算法自动分配。

From00 · 2025-01-09T03:11:28Z

python/paddle/distributed/auto_parallel/static/pir_pass.py

+        pp_stage_layer_nums = pp_stage_layer_num[pp_stage]
+        for i in range(0, pp_stage_layer_nums):
+            # The pp_stage uses a Round robin scheduling algorithm to allocate layers one by one.
+            v_chunk_id = i % vpp_degree


v和r是什么的缩写？

一个是virtual虚拟chunk_id，一个是real真实chunk_id

… pipeline_vpp

From00

LGTM

支持非均衡VPP编排的灵活模型层分配策略

23694c1

paddle-bot bot added the contributor External developers label Dec 14, 2024

zty-king added 3 commits December 15, 2024 08:27

支持非均衡VPP编排的灵活模型层分配策略

a067477

支持非均衡VPP编排的灵活模型层分配策略

e3e1e71

支持非均衡VPP编排的灵活模型层分配策略

708b467

From00 reviewed Dec 16, 2024

View reviewed changes

zty-king added 3 commits December 16, 2024 09:18

支持非均衡VPP编排的灵活模型层分配策略

abd297a

支持非均衡VPP编排的灵活模型层分配策略

b4cc018

支持非均衡VPP编排的灵活模型层分配策略

4f7e586

From00 reviewed Dec 24, 2024

View reviewed changes

zty-king added 2 commits January 8, 2025 08:03

支持非均衡VPP编排的灵活模型层分配策略

b0d9ac9

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

30f543a

… pipeline_vpp

From00 reviewed Jan 9, 2025

View reviewed changes

zty-king added 2 commits January 9, 2025 05:05

支持非均衡VPP编排的灵活模型层分配策略

5c6d235

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

94a0d32

… pipeline_vpp

From00 approved these changes Jan 9, 2025

View reviewed changes

From00 merged commit 266e3cd into PaddlePaddle:develop Jan 9, 2025
30 of 31 checks passed

zty-king mentioned this pull request Jan 14, 2025

2024下半年飞桨开源之星评选-信息征集 PaddlePaddle/community#1043

Closed

	pp_stage_layer_nums = pp_stage_layer_num[pp_stage]
	pp_stage_layer_num = pp_stage_layer_nums[pp_stage]

	seg_layer_num = [0] * num_chunks
	# Each chunk is assigned a number of layers based on user intent.
	seg_layer_num = [0] * num_chunks

支持非均衡VPP编排的灵活模型层分配策略 #70230

支持非均衡VPP编排的灵活模型层分配策略 #70230

Uh oh!

Conversation

zty-king commented Dec 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Category

PR Types

Description

Uh oh!

paddle-bot bot commented Dec 14, 2024

Uh oh!

From00 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paddle-ci-bot bot commented Dec 28, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

From00 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zty-king commented Dec 14, 2024 •

edited

Loading