Compile time - runtime separation / single node multiple GPU milestones

[helinwang]

1. Attributes use Attr proto message only (any C++ trick to make changing to it easier?):
   The current attribute items can't be serialized, we can't use ProgramDesc for executor unless we change this.

   $ cd paddle/paddle
   
   # brew install ack, or use ack-grep on linux
   $ ack 'Attr<' |sed 's/Attr</|/' |awk -F'|' '{print $2}'|sed 's/>(/|/' |awk -F'|' '{print $1}'|sort|uniq
   AttrType
   T
   bool
   float
   framework::BlockDescBind *
   int
   int32_t
   size_t
   std::string
   std::vector<int>
   std::vector<std::string>
   # not a single OpDesc::Attr
   

2. C++ Executor take ProgramDesc

   Eventually we will change to ExecutionPlan, ProgramDesc is similar to ExecutionPlan (both can be serialized), but ProgramDescBind is too different from ExecutionPlan. If we develop multiple thread executor based on ProgramDescBind, there are too much to change later.

3. ExecutionPlan design doc (Add ExecutionPlan design. #6078)

4. change C++ Executor to take ExecutionPlan

5. Multiple thread executor design doc

6. C++ multiple thread Executor

7. simple C++ planner

   Just place everything on CPU / GPU-0

8. Modular Python Executor

   current code:

   # pseudo code
   # implement local Python executor first, remote Python executor later.
   def run(program):
     plan = planner.plan(program, local_devices)
     fetch_vars = cpp_executor.run(plan, g_scope)
     return fetch_vars

9. single node multiple GPU planner

Dependency

[jacquesqiao]

Attr proto message: https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/framework.proto#L75

https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/framework.proto#L19

[jacquesqiao]

https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/while_op.cc#L49
this line use attr to store a blockdescbind pointer and have serialize problem

[typhoonzero]

https://stackoverflow.com/questions/13603878/how-to-delete-arbitrary-objects-in-repeated-field-protobuf

We don't have method of removing objects from protobuf

[jacquesqiao]

so we need some helper function to help modify the execution plan proto message.

[reyoung]

The current attribute items can't be serialized, we can't use ProgramDesc for executor unless we change this.

The current attribute can be serialized. https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/op_desc.cc#L296

[Yancey1989]

Need a Parallel Operator to execute the block with multi threads.

[helinwang]

This issue is out-of-date, closing.