PR types

Function optimization

PR changes

Others

Description

Pcard-75624

Fix the ambiguity of node execution event, the first segment is for calling GradNodeFunction, should be recognized as XXXGradNode computation cost, the second segment is for potential gradient accumulation in backward queue, should not be count into XXXGradNode, or will misleading users who profile paddle program with nsight or other visualization software.

So this PR use Local_XXXGradNode to represent execution time of XXXGradNode function, Global_XXXGradNode to represent execution time of Local_XXXGradNode plus potential gradient accumulation. Thus, Global_XXXGradNode should always be larger than Local_XXXGradNode and Local_XXXGradNode is more significant for profiling.

To achieve this target, this PR modifies eager_gen.py and several manually XXXnode.cc, and move event creation next to node execution for ignoring node(s) skipped in backward node queue(i.e.

).

before:

after:

As is dipicted below, Global_MultiplyGradNode include 2 parts: Local_MultiplyGradNode execution(grad_output_tensors = (*node)( node_input_buffer->Buffers(), create_graph, is_general_grad);) and gradient accumulation(node_input_buffers_dict[next_node]->add(edge_rank.first, edge_rank.second, grad_output_tensor, create_graph);)