[Kunlun] cherry-pick:fix multi xpu dygraph hang, test=kunlun

[vslyu]

PR types

Bug fixes

PR changes

Others

Describe

cherry-pick [PR32662]:#32662
Fix multi kunlun XPU cards dygraph running hang.
ref to PaddleClas [PR690]:PaddlePaddle/PaddleClas#690

python3.7 -m paddle.distributed.launch --xpus=2,3 --log_dir log tools/train.py -c ./configs/quick_start/ResNet50_vd_finetune_kunlun.yaml

2021-04-29 08:21:19,988 - INFO - epoch:19 , train step:0   , top1: 0.93750, top5: 1.00000, loss: 0.92145, lr: 0.000023, batch_cost: 0.96966 s, reader_cost: 0.23445 s, ips: 33.00113 images/sec, eta: 0:00:14
2021-04-29 08:21:26,911 - INFO - epoch:19 , train step:10  , top1: 0.93750, top5: 1.00000, loss: 1.03136, lr: 0.000003, batch_cost: 0.69550 s, reader_cost: 0.00019 s, ips: 46.01037 images/sec, eta: 0:00:03
2021-04-29 08:21:29,695 - INFO - END epoch:19  train top1: 0.92292, top5: 0.98542, loss: 0.95281,  batch_cost: 0.69007 s, reader_cost: 0.00025 s, batch_cost_sum: 3.45033 s, ips: 46.37236 images/sec.

[paddle-bot-old]

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.