PR types

New features

PR changes

Others

Description

Pcard-72023

The optimal configuration for large model with distributed training/inference often requires designing multiple sets of experiments based on experiences (network, parameter size, gpu memory or flops, etc.), and comparing the results to determine the optimal configuration. This process heavily relies on human experience, and the determined optimal configuration may not be the global optimal configuration. When any condition changes, the above process needs to be repeated repeatedly, resulting in poor usability of large models.

To address the above issues, we have implemented AutoTuner based on Profiling, with the main modules as follows:

1. Provide clear json configuration for users to directly use AutoTuner, avoiding additional coding work for users
2. launch multi tasks one by one and automatically schedule and monitor.
3. Implement search module and pruning module, support multiple search algorithms and pruning strategies.

At present, we have built-in grid search support for 8 dimensions, including dp degree, mp degree, pp degree, mbs, sharding degree, sharding stage, recompute, and recompute granularity. The example JSON is as follows:

The usage is as follows:
python -m paddle.distributed.launch --devices "0,1,2,3,4,5,6,7" --auto_tuner_json=test.json your_train.py your_args

NOTE: Since the auto_tuner is non-invasive, users need to expose args in their script to enable the configuration generated by auto_tuner be executed.