[Auto Parallel] Improve the fine-grained APIs

[aoyulong]

PR types

Others

PR changes

APIs

Describe

In the pr, we proposed a fine-grained APIs to satisfy users who want to control the execution logic. This pr is used to further improve these fine-grained APIs as following:

• Improved Distributed loader
  • Add distributed dataloader (recommended) and dataloader_from_generator (for legacy support) methods.
  • Align the interface to the serial dataloader as close as possible.
• Improved Fine-grained APIs: dataloader + prepare + run

  import paddle
  import paddle.vision.transforms as T
  import paddle.distributed.auto_parallel as auto
  from paddle.vision.datasets import MNIST
  
  transform = T.Compose([
     T.Transpose(),
     T.Normalize([127.5], [127.5])
  ])
  train_dataset = MNIST(mode='train', transform=transform)
  valid_dataset = MNIST(mode='test', transform=transform)
  
  model = paddle.vision.models.LeNet()
  loss = paddle.nn.CrossEntropyLoss() 
  optimizer = paddle.optimizer.Adam(
     learning_rate=0.001, parameters=model.parameters())
  metrics = paddle.metric.Accuracy(topk=(1, 2))
  
  engine = auto.Engine(model, loss, optimizer, metrics)
  
  # Step 1: build distributed dataloader
  dataloader = engine.dataloader(train_dataset,
                                epochs=2, 
                                batch_size=64, 
                                mode="train")
  
  # Step 2: build the distributed program
  engine.prepare(mode="train")
  
  feed_dict = ...
  fetch_list = ...
  # Step 3: run the distributed program by using the distributed dataloader
   for data in train_dataloader:
       outs = engine.run(data, feed=feed_dict, fetch_list=fetch_list, mode="train")

  • Add the prepare API to explicitly control the partition.
  • Replace __call__ with run to conform with the serial executor interface.

[paddle-bot]

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[JZ-LIANG]

LGTM