Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[hybrid performance] Optimize Pipeline Scheduler #35680

Merged
merged 8 commits into from
Sep 14, 2021
Merged

[hybrid performance] Optimize Pipeline Scheduler #35680

merged 8 commits into from
Sep 14, 2021

Conversation

FeixLiu
Copy link
Contributor

@FeixLiu FeixLiu commented Sep 13, 2021 •
edited
Loading

PR types

Performance optimization

PR changes

Others

Describe

Update the pipeline scheduler, move the forward/backward/optimize op decision logic to init function of SectionWorker.

Loss compare on GPU with Ernie3.0:
image

Speed:

Before After Gain
35358 36032 +1.9%

@paddle-bot-old
Copy link

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

wangxicoding
wangxicoding requested changes Sep 13, 2021
View reviewed changes
paddle/fluid/framework/section_worker.cc Outdated Show resolved Hide resolved
paddle/fluid/framework/device_worker.h Outdated Show resolved Hide resolved
paddle/fluid/framework/device_worker.h Outdated Show resolved Hide resolved
wangxicoding
wangxicoding previously approved these changes Sep 13, 2021
View reviewed changes
Copy link
Contributor

@wangxicoding wangxicoding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

wangxicoding
wangxicoding approved these changes Sep 14, 2021
View reviewed changes
Copy link
Contributor

@wangxicoding wangxicoding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

sandyhouse
sandyhouse reviewed Sep 14, 2021
View reviewed changes
Copy link

@sandyhouse sandyhouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

从该pr描述部分看，该pr的loss曲线在末端会高于基线。该pr应该不改变调度逻辑，为什么会引起loss曲线的变化？

@FeixLiu
Copy link
Contributor Author

FeixLiu commented Sep 14, 2021

从该pr描述部分看，该pr的loss曲线在末端会高于基线。该pr应该不改变调度逻辑，为什么会引起loss曲线的变化？

随机初始化带来的吧
Screen Shot 2021-09-14 at 1 52 54 PM
用了一个另一次跑的baseline做的新对比图

sandyhouse
sandyhouse approved these changes Sep 14, 2021
View reviewed changes
Copy link

@sandyhouse sandyhouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wangxicoding wangxicoding merged commit 04fdb10 into PaddlePaddle:develop Sep 14, 2021
@FeixLiu FeixLiu deleted the pipeline_schedule branch September 14, 2021 06:00
AnnaTrainingG pushed a commit to AnnaTrainingG/Paddle that referenced this pull request Sep 29, 2021
@FeixLiu @AnnaTrainingG
[hybrid performance] Optimize Pipeline Scheduler (PaddlePaddle#35680)
0a04a0f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sandyhouse sandyhouse sandyhouse approved these changes

@wangxicoding wangxicoding wangxicoding approved these changes

@gongweibao gongweibao Awaiting requested review from gongweibao

@fuyinno4 fuyinno4 Awaiting requested review from fuyinno4

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@FeixLiu @sandyhouse @wangxicoding