Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added oneDNN reduce_op GRAD kernel #32280

Merged
merged 43 commits into from
Apr 21, 2021
Merged

Added oneDNN reduce_op GRAD kernel #32280

merged 43 commits into from
Apr 21, 2021

Conversation

jakpiase
Copy link
Contributor

PR types

New features

PR changes

OPs

Describe

Added oneDNN reduce_op GRAD fp32 and bf16 kernels (reduce_sum, reduce_mean) for enabling Word2Vec model.
Forward operator PR link: Reduce FWD

Jakub Piasecki and others added 29 commits December 2, 2020 14:40
added external reorder to profiler
1ecc4cf
@jakpiase
resolved conflicts
d4f9ad4
@jakpiase
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
f85e7a3 
… reduce_onednn
@jakpiase
added mkldnn reduce op kernel
5c02f89
@jakpiase
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
7c3b736 
… reduce_onednn
@jakpiase
refactored reduce op
4147b25
@jakpiase
reverted old file
726846f
@jakpiase
added clang formatting
6763404
@jakpiase
removed unnecessary imports and comments
f2555e5
@jakpiase
minor change
8f80eb5
@jakpiase
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
539fe3c 
… reduce_onednn
@jakpiase
merged with develop
3dfabd9
@jakpiase
Revert "merged with develop"
895f948 
This reverts commit 3dfabd9.
@jakpiase
minor change
cd9d2f3
@jakpiase
fixed mispelling
87fc5a1
@jakpiase
Minor refactoring
a75ee12
@jakpiase
minor change
b442889
@jakpiase
importet necessary modules
27dec3a
@jakpiase
minor change
71089fe
@jakpiase
minor formatting change
29097ce
@jakpiase
excluded cuda from bf test
164043a
@jakpiase
fixed static mode in test_resnet_v2
be36f94
@jakpiase
added formatting
424083f
@jakpiase
added support for edge case
87b5b38
@jakpiase
added files for reduce grad
94e4ace
@jakpiase
added grad tests for onednn reduce
9ae1005
@jakpiase
resolved conflicts
cfa2519
@jakpiase
added formatting
7d3797f
@jakpiase
minor changes
782e25c
@paddle-bot-old
Copy link

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@jakpiase
Copy link
Contributor Author

@jczaja @arogowie-intel Could you please review this code?

arogowie-intel
arogowie-intel reviewed Apr 15, 2021
View reviewed changes
paddle/fluid/operators/reduce_ops/mkldnn/reduce_mkldnn_op.h Outdated
case 4:
return mkldnn::memory::format_tag::abcd;
default:
return mkldnn::memory::format_tag::abcde;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why 5 dim tensor is a default case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have made a restriction in GetExpectedKernelType that dims must be in range <1,5>. I had to ensure the compiler that there always will be a return value from this function. I can delete the default statement and just leave the instruction outside switch block. What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add case 5: and in default statement throw an error that invalid argument passed.

arogowie-intel
arogowie-intel suggested changes Apr 16, 2021
View reviewed changes
paddle/fluid/operators/reduce_ops/mkldnn/reduce_mkldnn_op.h Outdated
case 4:
return mkldnn::memory::format_tag::abcd;
default:
return mkldnn::memory::format_tag::abcde;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd add case 5: and in default statement throw an error that invalid argument passed.

paddle/fluid/operators/reduce_ops/mkldnn/reduce_mkldnn_op.h Outdated
void RunKernel(const framework::ExecutionContext& ctx,
dnnl::algorithm binary_type, float scale_x,
float scale_y) const {
auto& dev_ctx =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since you're not modifying dev_ctx:

Suggested change
auto& dev_ctx =
const auto& dev_ctx =

paddle/fluid/operators/reduce_ops/mkldnn/reduce_mkldnn_op.h Outdated

auto dims = ctx.Attr<std::vector<int>>("dim");
auto* input_dy = ctx.Input<Tensor>(framework::GradVarName("Out"));

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this blank line.

Suggested change

paddle/fluid/operators/reduce_ops/mkldnn/reduce_mkldnn_op.h Outdated
auto* output_dx = ctx.Output<Tensor>(framework::GradVarName("X"));

output_dx->mutable_data<T>(ctx.GetPlace());

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

paddle/fluid/operators/reduce_ops/mkldnn/reduce_mkldnn_op.h Outdated

auto src_dx_memory = handler.AcquireSrcMemory(output_dx);
const auto src_dy_memory = handler.AcquireSecondSrcMemory(input_dy);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

python/paddle/fluid/tests/unittests/mkldnn/test_reduce_mkldnn_op.py Outdated
@@ -83,8 +71,8 @@ def setUp(self):
}


@skip_check_grad_ci(reason="not implemented")
class TestReduceSum5DReduceAllKeepDimsONEDNNOp(TestReduceSumDefaultONEDNNOp):
class TestReduceSum5DReduceAllKeepDimsONEDNNOp(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
class TestReduceSum5DReduceAllKeepDimsONEDNNOp(
class TestReduceSum5DReduceAllKeepDimsOneDNNOp(

python/paddle/fluid/tests/unittests/mkldnn/test_reduce_mkldnn_op.py Outdated
@skip_check_grad_ci(reason="not implemented")
class TestReduceSum5DReduceAllKeepDimsONEDNNOp(TestReduceSumDefaultONEDNNOp):
class TestReduceSum5DReduceAllKeepDimsONEDNNOp(
TestReduceDefaultWithGradONEDNNOp):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
TestReduceDefaultWithGradONEDNNOp):
TestReduceDefaultWithGradOneDNNOp):

python/paddle/fluid/tests/unittests/mkldnn/test_reduce_mkldnn_op.py Outdated
@@ -95,8 +83,7 @@ def setUp(self):
}


@skip_check_grad_ci(reason="not implemented")
class TestReduceSum4DReduceAllONEDNNOp(TestReduceSumDefaultONEDNNOp):
class TestReduceSum4DReduceAllONEDNNOp(TestReduceDefaultWithGradONEDNNOp):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
class TestReduceSum4DReduceAllONEDNNOp(TestReduceDefaultWithGradONEDNNOp):
class TestReduceSum4DReduceAllOneDNNOp(TestReduceDefaultWithGradOneDNNOp):

python/paddle/fluid/tests/unittests/mkldnn/test_reduce_mkldnn_op.py Outdated
@@ -154,8 +141,7 @@ def setUp(self):
}


@skip_check_grad_ci(reason="not implemented")
class TestReduceMean3DONEDNNOp(TestReduceSumDefaultONEDNNOp):
class TestReduceMean3DONEDNNOp(TestReduceDefaultWithGradONEDNNOp):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
class TestReduceMean3DONEDNNOp(TestReduceDefaultWithGradONEDNNOp):
class TestReduceMean3DOneDNNOp(TestReduceDefaultWithGradOneDNNOp):

python/paddle/fluid/tests/unittests/mkldnn/test_reduce_mkldnn_op.py Outdated
@@ -166,8 +152,7 @@ def setUp(self):
}


@skip_check_grad_ci(reason="not implemented")
class TestReduceMean4DReduceAllONEDNNOp(TestReduceSumDefaultONEDNNOp):
class TestReduceMean4DReduceAllONEDNNOp(TestReduceDefaultWithGradONEDNNOp):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
class TestReduceMean4DReduceAllONEDNNOp(TestReduceDefaultWithGradONEDNNOp):
class TestReduceMean4DReduceAllOneDNNOp(TestReduceDefaultWithGradOneDNNOp):

jczaja
jczaja approved these changes Apr 19, 2021
View reviewed changes
Copy link
Contributor

@jczaja jczaja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jakpiase
Copy link
Contributor Author

@arogowie-intel I have implemented all your suggestions except one. Could you please re-review?

arogowie-intel
arogowie-intel approved these changes Apr 20, 2021
View reviewed changes
Copy link
Contributor

@arogowie-intel arogowie-intel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job!

@jakpiase
Copy link
Contributor Author

@luotao1 Could you please start your review?

@luotao1 luotao1 merged commit ead8342 into PaddlePaddle:develop Apr 21, 2021
@lidanqing-intel lidanqing-intel mentioned this pull request Apr 22, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@luotao1 luotao1 luotao1 approved these changes

@jczaja jczaja jczaja approved these changes

@arogowie-intel arogowie-intel arogowie-intel approved these changes

Assignees

@luotao1 luotao1

Labels
Intel
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jakpiase @luotao1 @jczaja @arogowie-intel @lidanqing-intel