[bug fix] fix the allreduce fused bug

[FeixLiu]

PR types

Bug fixes

PR changes

Others

Describe

Rewrite the logic for allreduce fuse under raw_program_optimizer.

Test performance and precision based on MOE model on 8 A100 GPUs.

Performance gain

Without fuse throughput	Fuse_num = 8 throughput	Performance gain
5.213 steps / sec	5.453 steps / sec	+4.6%

Loss curve

Loss diff

max loss diff: 0.2944
min loss diff: 0.0001

How to use

strategy = paddle.distributed.fleet.DistributedStrategy()
strategy.without_graph_optimization = True
strategy.fuse_all_reduce_ops = True
# the following two flags are optional
strategy._calc_comm_same_stream = False    # use calc stream as comm stream or not, default is False
strategy.fuse_grad_size_in_num = 8         # the number of grad in one allreduce_sum op, default is 8

[paddle-bot-old]

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[wangxicoding]

LGTM