[AMP]split minimize and add unscale_ for GradScaler

[zhangbo9674]

PR types

New features

PR changes

APIs

Describe

1、Split function GradScaler::minimize() to GradScaler::step() + GradScaler::update()：

GradScaler::minimize():

    scaler = paddle.amp.GradScaler(init_loss_scaling=1024)

    with paddle.amp.auto_cast():
        output = model(data)
        loss = mse(output, label)

    scaled = scaler.scale(loss)
    scaled.backward()            
    scaler.minimize(optimizer, scaled)
    optimizer.clear_grad()

GradScaler::step() + GradScaler::update():

    scaler = paddle.amp.GradScaler(init_loss_scaling=1024)

    with paddle.amp.auto_cast():
        output = model(data)
        loss = mse(output, label)

    scaled = scaler.scale(loss)
    scaled.backward() 
    scaler.step(optimizer)
    scaler.update()
    optimizer.clear_grad()

• minimize() and step()+update() are two methods of parameter gradient updating in amp. In paddle 2.0, we recommend using step()+update().

• If optimizer belongs to paddle 1.0, only minimize() can be used.

2、Add GradScaler::unscale_(optimizer):

    scaler = paddle.amp.GradScaler(init_loss_scaling=1024)

    with paddle.amp.auto_cast():
        output = model(data)
        loss = mse(output, label)

    scaled = scaler.scale(loss)
    scaled.backward() 
    scaled.unscale_(optimizer)
    scaler.step(optimizer)
    scaler.update()
    optimizer.clear_grad()

• This API is used to unscale the gradients of parameters, multiplies the gradients of parameters by 1/(loss scaling ratio).
• If unscale_ is not called, minimize() or step() will call this api, else this call will not be repeated.

3、docs review:

GradScaler:

step+update:

unscale_:

中文文档pr链接：
PaddlePaddle/docs#3897

[paddle-bot-old]

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[zhiqiu]

LGTM