clear local scope every setp #37569
Merged
clear local scope every setp #37569
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
zhiqiu
reviewed
Nov 26, 2021
| // return Fetch Tensors | ||
| auto* fetch_var = global_scope_->Var(interpreter::kFetchVarName); | ||
| return std::move(*fetch_var->GetMutable<framework::FetchList>()); | ||
| } | ||
|
|
||
| void InterpreterCore::ClearLocalScope() { |
There was a problem hiding this comment.
Rename to ClearLoDTensorArrayInLocalScope and add some comments
Zjq9409
pushed a commit
to Zjq9409/Paddle
that referenced
this pull request
Dec 10, 2021
* clear local scope every setp, test=develop * refine,test=develop * refine, test=develop
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Bug fixes
PR changes
Others
Describe
PR #37474 在解决
test_transformer单测时，在GC中clear了 LoDTensorArray。这导致test_slice_op单测失败：test_transformer单测遇到的问题是，第一轮运行完后，某LoDTensorArray Tensors中的Holder被GC回收了，而LoDTensorArray的vector中仍保留着原有的几个没有Holder的Tensor。而在第二轮时，算子直接在原有LoDTensorArray基础上push_back了几个新的Tensor，此时，前几个Tensor的Holder是为空的。进而导致了后续运行中出现错误。test_slice_op单测中发生如下场景：slice的输入为一个LoDTensorArray，同时这个输入也是slice_grad的输入，但slice_grad对这个LoDTensorArray的要求是NoNeedBuffer的。slice_grad只需要这个LoDTensorArray的size。因此在slice运行结束后就GC了LoDTensorArray，并clear了vector。而到slice_grad运行时，则无法拿到LoDTensorArray的size。在原Executor是通过在step结束时，调用DeleteScope会清理掉所有的Variable，因此不会有该问题。
原PE中，也是通过设置
num_iteration_per_drop_scope清理scope时，我猜测，在test_transformer这种场景下num_iteration_per_drop_scope的值必须为1。目前本PR的做法是，在每个Step结束，如果local scope中的Var是LoDTensorArray类型，则调用clear。
需要做GC检查吗？