Add some passes which can be applied to Program #34730

sneaxiy · 2021-08-09T10:51:32Z

PR types

New features

PR changes

Others

Describe

Add some passes which can be applied to Program:

sync_batch_norm_pass
fuse_relu_depthwise_conv_pass
fuse_bn_act_pass
fuse_bn_add_act_pass
fusion_group_pass
fuse_elewise_add_act_pass
fuse_adam_op_pass
fuse_sgd_op_pass
fuse_momentum_op_pass
runtime_context_cache_pass
inplace_addto_op_pass
buffer_shared_inplace_pass

TODO: add fuse_allreduce_op_pass.

paddle-bot-old · 2021-08-09T10:51:35Z

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

fix compile error of op compat

chenwhql

LGTM for framework.py & blcok_desc.h

chenwhql · 2021-08-10T11:35:37Z

paddle/fluid/framework/block_desc.cc

@@ -238,5 +238,37 @@ BlockDesc *BlockDesc::ForwardBlock() const {
  return prog_->MutableBlock(static_cast<size_t>(desc_->forward_block_idx()));
 }

+void BlockDesc::MoveFrom(BlockDesc *block) {
+  PADDLE_ENFORCE_NOT_NULL(
+      block, platform::errors::InvalidArgument("block must be provided"));


we recommend the first letter of error message sentence capitalized and ends with ., other cases are same

zhiqiu · 2021-08-12T13:07:39Z

paddle/fluid/operators/share_buffer_op.h

+      outputs[i]->ShareBufferWith(*inputs[i]);
+      VLOG(10) << "Share tensor buffer " << (*input_args)[i] << " -> "
+               << (*output_args)[i];
+      if (!share_dims.empty() && share_dims[i]) {


Better check share_dims.size()==n

Have been checked here.

sneaxiy · 2021-08-13T04:15:01Z

python/paddle/fluid/ir.py

+        update_attr(attrs, attr_types, "nranks", 1, "size_t")
+        update_attr(attrs, attr_types, "use_cuda", False, "bool")
+        # TODO(zjl): how to skip fetch variables ?
+        update_attr(attrs, attr_types, "mem_opt_skip_vars",


No. I only skip the variables that satisfies var.is_data == True currently.

zhiqiu · 2021-08-12T13:28:04Z

paddle/fluid/framework/ir/pass.cc

+    const auto &startups =
+        graph.Get<details::ProgramDescs>(details::kStartupProgramDescs);
+    VLOG(10) << "Merge startup programs";
+    MergePrograms(startup_program, startups, /*append=*/true);


I wonder why we need to merge here? IF startup_program is the applied program, why it will lose some vars or ops?

Commonly, startup_program would not lose some vars or ops. We only need to merge the created kStartupProgramDescs to the startup_program. Some passes may produce kStartupProgramDescs (a list of some ProgramDescs) (see here) and run it once when using ParallelExecutor (see here). So when we apply passes to ProgramDescs, startup_program should contain these kStartupProgramDescs, so that these new created ops can run and only run once.

zhiqiu · 2021-08-12T13:39:39Z

paddle/fluid/framework/ir/memory_optimize_pass/inplace_addto_op_pass.cc

+    if (is_first_var_valid == is_second_var_valid) {
+      continue;
+    }


is_first_var_valid and is_second_var_valid are both true is ok ?

Yes. I agree with you.

sneaxiy

Thanks for your suggestions, @zhiqiu . I do think is_first_var_valid and is_second_var_valid are both true is ok. I have added the logic. Please review again.

sneaxiy · 2021-08-13T04:13:45Z

paddle/fluid/operators/share_buffer_op.h

+      outputs[i]->ShareBufferWith(*inputs[i]);
+      VLOG(10) << "Share tensor buffer " << (*input_args)[i] << " -> "
+               << (*output_args)[i];
+      if (!share_dims.empty() && share_dims[i]) {


Have been checked here.

sneaxiy · 2021-08-13T04:15:01Z

python/paddle/fluid/ir.py

+        update_attr(attrs, attr_types, "nranks", 1, "size_t")
+        update_attr(attrs, attr_types, "use_cuda", False, "bool")
+        # TODO(zjl): how to skip fetch variables ?
+        update_attr(attrs, attr_types, "mem_opt_skip_vars",


No. I only skip the variables that satisfies var.is_data == True currently.

sneaxiy · 2021-08-13T04:18:42Z

paddle/fluid/framework/ir/pass.cc

+    const auto &startups =
+        graph.Get<details::ProgramDescs>(details::kStartupProgramDescs);
+    VLOG(10) << "Merge startup programs";
+    MergePrograms(startup_program, startups, /*append=*/true);


Commonly, startup_program would not lose some vars or ops. We only need to merge the created kStartupProgramDescs to the startup_program. Some passes may produce kStartupProgramDescs (a list of some ProgramDescs) (see here) and run it once when using ParallelExecutor (see here). So when we apply passes to ProgramDescs, startup_program should contain these kStartupProgramDescs, so that these new created ops can run and only run once.

zhiqiu · 2021-08-13T06:40:41Z

python/paddle/fluid/tests/unittests/test_apply_pass_to_program.py

+
+        scope1 = paddle.static.Scope()
+        with paddle.static.scope_guard(scope1):
+            self.executor.run(startup1)
+
+        scope2 = paddle.static.Scope()
+        with paddle.static.scope_guard(scope2):
+            self.executor.run(startup2)


Suggested change

scope1 = paddle.static.Scope()

with paddle.static.scope_guard(scope1):

self.executor.run(startup1)

scope2 = paddle.static.Scope()

with paddle.static.scope_guard(scope2):

self.executor.run(startup2)

paddle.seed(2021)

scope1 = paddle.static.Scope()

with paddle.static.scope_guard(scope1):

self.executor.run(startup1)

paddle.seed(2021)

scope2 = paddle.static.Scope()

with paddle.static.scope_guard(scope2):

self.executor.run(startup2)

Done. Thanks for you suggestions!

zhiqiu

LGTM

zhhsplendid · 2021-08-13T07:03:23Z

paddle/fluid/framework/ir/memory_optimize_pass/buffer_shared_inplace_op_pass.cc

+  PADDLE_ENFORCE_EQ(block.ID(), 0, platform::errors::Unimplemented(
+                                       "Inplace can only perform in block 0."));
+  // only take block 0 gc_vars
+  const auto all_gc_vars =


Readability: suggest to rename 'all_gc_vars' to 'op_gc_vars' or other name, then it is easier to know why its size is same to all_ops.size() and we can know 'op_gc_vars[i]' means 'gc_vars' of 'op[i]'

zhhsplendid · 2021-08-13T07:16:12Z

paddle/fluid/framework/ir/memory_optimize_pass/buffer_shared_inplace_op_pass.cc

+    for (const auto &pair : var_pair) {
+      const auto &input_slot = pair.first;
+      const auto &output_slot = pair.second;
+      auto input_var = GetFirstVarName(op, input_slot, true);


It is not forward to read the meaning of boolean without looking at the code of GetFirstVarName.

Suggest for two options:

`/* is_input = */ true

2 Change GetFirstVarName to GetNameMapFirstVar, then pass it op.Inputs() instead of op

lanxianghit

API.spec changed，but no Python API changed，approve for this.

jzhang533

lgtm

Xreki

LGTM for op benchmark ci

zhhsplendid

LGTM

sneaxiy added 3 commits August 9, 2021 10:41

add inplace passes and tests

3fa2507

Merge upstream/develop

ae7cf21

update

8f913ac

sneaxiy added 6 commits August 9, 2021 11:16

fix use_cuda undefined

9606931

fix compile error of op compat

add more ut

6545a62

fix CPU CI error

a705a4f

check adam unique

81b1e82

fix mac/windows ci, improve coverage

0a8586c

fix ci error

5329b0f

sneaxiy requested review from zhhsplendid, zhiqiu, chenwhql, dingjiaweiww and Xreki August 10, 2021 09:53

chenwhql reviewed Aug 10, 2021

View reviewed changes

sneaxiy added 2 commits August 10, 2021 14:02

follow weihang's comment

466d483

fix BlockDesc::MoveFrom

94ba44f

zhiqiu reviewed Aug 12, 2021

View reviewed changes

follow qiuliang's comment

2da6a3a

sneaxiy commented Aug 13, 2021

View reviewed changes

zhiqiu reviewed Aug 13, 2021

View reviewed changes

update

3851114

zhiqiu previously approved these changes Aug 15, 2021

View reviewed changes

zhhsplendid reviewed Aug 16, 2021

View reviewed changes

follow huihuang's comments

79525d1

sneaxiy dismissed zhiqiu’s stale review via 79525d1 August 16, 2021 06:28

sneaxiy requested review from XiaoguangHu01, lanxianghit and jzhang533 August 16, 2021 09:16

zhiqiu approved these changes Aug 16, 2021

View reviewed changes

lanxianghit approved these changes Aug 16, 2021

View reviewed changes

jzhang533 approved these changes Aug 16, 2021

View reviewed changes

Xreki approved these changes Aug 17, 2021

View reviewed changes

zhhsplendid approved these changes Aug 17, 2021

View reviewed changes

sneaxiy merged commit 8046e33 into PaddlePaddle:develop Aug 17, 2021

sneaxiy deleted the program_pass_dev branch August 17, 2021 03:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add some passes which can be applied to Program #34730

Add some passes which can be applied to Program #34730

sneaxiy commented Aug 9, 2021 •

edited

Loading

paddle-bot-old bot commented Aug 9, 2021

chenwhql left a comment

chenwhql Aug 10, 2021

sneaxiy Aug 10, 2021

zhiqiu Aug 12, 2021

sneaxiy Aug 13, 2021

sneaxiy Aug 13, 2021

zhiqiu Aug 12, 2021

sneaxiy Aug 13, 2021

zhiqiu Aug 12, 2021

sneaxiy Aug 13, 2021

sneaxiy left a comment

sneaxiy Aug 13, 2021

sneaxiy Aug 13, 2021

sneaxiy Aug 13, 2021

zhiqiu Aug 13, 2021

sneaxiy Aug 13, 2021

zhiqiu left a comment

zhhsplendid Aug 13, 2021

sneaxiy Aug 16, 2021

zhhsplendid Aug 13, 2021

sneaxiy Aug 16, 2021

lanxianghit left a comment

jzhang533 left a comment

Xreki left a comment

zhhsplendid left a comment

Add some passes which can be applied to Program #34730

Add some passes which can be applied to Program #34730

Conversation

sneaxiy commented Aug 9, 2021 • edited Loading

PR types

PR changes

Describe

paddle-bot-old bot commented Aug 9, 2021

chenwhql left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sneaxiy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhiqiu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lanxianghit left a comment

Choose a reason for hiding this comment

jzhang533 left a comment

Choose a reason for hiding this comment

Xreki left a comment

Choose a reason for hiding this comment

zhhsplendid left a comment

Choose a reason for hiding this comment

sneaxiy commented Aug 9, 2021 •

edited

Loading