Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[hybrid performance] pipeline cache trainer #33998

Merged
merged 2 commits into from
Jul 9, 2021
Merged

[hybrid performance] pipeline cache trainer #33998

merged 2 commits into from
Jul 9, 2021

Conversation

FeixLiu
Copy link
Contributor

@FeixLiu FeixLiu commented Jul 7, 2021 •
edited by wangxicoding
Loading

PR types

Performance optimization

PR changes

Others

Describe

program cache for PP

V100 32G，gpt2-en模型测试

卡数 优化 dtype speed(tokens/s) （S） 提升
4卡pp baseline fp32 21951
fp16 43094
4卡pp 干掉_dump_debug_info fp32 24174 10.1%
fp16 60900 41.3%

@paddle-bot-old
Copy link

paddle-bot-old bot commented Jul 7, 2021

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@FeixLiu FeixLiu force-pushed the program_cache branch 3 times, most recently from 7607cdb to 99c4571 Compare July 7, 2021 08:33
wangxicoding
wangxicoding reviewed Jul 9, 2021
View reviewed changes
python/paddle/fluid/executor.py

dataset._dynamic_adjust_after_train()
dataset._finish_to_run()
if not use_program_cache:
Copy link
Contributor

@wangxicoding wangxicoding Jul 9, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

后面可以加个注释

wangxicoding
wangxicoding previously approved these changes Jul 9, 2021
View reviewed changes
Copy link
Contributor

@wangxicoding wangxicoding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

sandyhouse
sandyhouse approved these changes Jul 9, 2021
View reviewed changes
Copy link

@sandyhouse sandyhouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wangxicoding wangxicoding changed the title program cache startup [hybrid performance] program cache startup Jul 9, 2021
@wangxicoding wangxicoding changed the title [hybrid performance] program cache startup [hybrid performance] pipeline cache trainer Jul 9, 2021
@wangxicoding wangxicoding merged commit 98c7191 into PaddlePaddle:develop Jul 9, 2021
@FeixLiu FeixLiu deleted the program_cache branch July 13, 2021 05:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sandyhouse sandyhouse sandyhouse approved these changes

@wangxicoding wangxicoding wangxicoding left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@FeixLiu @sandyhouse @wangxicoding