PR types

New features

PR changes

Others

Description

PCard-78429
This PR adds a new feature that estimates the optimal performance of multi Data Parallelism (DP) based on single DP, allowing for quickly identifying superior parallel strategies in large cluster environments.
The specific steps are：

1. Building a Single DP Search Space.

2. It can automatically profile the performance of a single-lane data parallel strategy based on the resources of a given small cluster.

3. Then, it calculates the performance of multi-lane data parallel strategies in a larger cluster based on the communication bandwidth.

Refer to the given example json to use this feature for PaddleNLP llama:
{ ... "search_algo":{ "name": "dp_estimation", "estimated_num_gpus": 1024, "conversion":{ "step_time": "interval_runtime", "comm_bw": [100, 200], "amp": true, "model_size_b": 7 }, "sharding_overlap": { "sharding_mp": ["--sharding_parallel_config", "enable_stage1_overlap"], "sharding_pp": ["--pipeline_parallel_config", "enable_sharding_comm_overlap"], "sharding_mp_pp": ["--pipeline_parallel_config", "enable_sharding_comm_overlap"] } } ... }
and the result is shown as follows: