[HybridParallel]Support 1f1b for PipelineParallel #34483
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
wangxicoding
approved these changes
Aug 2, 2021
| paddle.autograd.backward( | ||
| self.scaler.scale(self.caches['outputs'][cache_id])) | ||
| input_tensor_grad = self._backward_step(input_tensor, output_tensor, | ||
| output_tensor_grad) |
There was a problem hiding this comment.
output_tensor和output_tensor_grad用完了，貌似可以先手工释放一下
There was a problem hiding this comment.
应该不能手动设置为None，让它释放吧。host端提前释放，可能device还没开始计算。
There was a problem hiding this comment.
没问题的，gpu kernel调度拿到了地址，运行时候不被覆盖及被别人覆盖就行，可以试试🌚
| paddle.distributed.send(dtype, dst=1, group=group) | ||
|
|
||
| def send_meta(self, tensor, group): | ||
| if isinstance(tensor, paddle.Tensor): |
There was a problem hiding this comment.
提个建议，在pipeline_parallel.py里也有一大堆isinstance(tensor, tuple)的逻辑，不如把单个的paddle.Tensor封装成tuple，统一走tuple的逻辑
There was a problem hiding this comment.
确实！这个后面重写代码的时候，可以改的优美一些。
python/paddle/distributed/fleet/meta_parallel/pp_utils/p2p_communication.py
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
New features
PR changes
Others
Describe
[HybridParallel]Support 1f1b for PipelineParallel
修改当前流水线并行的调度方式，采用更省显存的1f1b的调度方式,类似于Megatron的 https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/schedules.py。
具体的调度图如下：
GPT-117M模型，V100-32G，PP=8， mircrobatch=2