[Feature Request]: Expand --extra_package option to allow passing a directory containing the package and deprecate the --setup_file option

[tvalentyn]

What would you like to happen?

The --setup_file option is becoming a misnomer now that many packages are defined with pyproject.toml. I think we can simplify the API and the documentation by sticking with the --extra_package option and allow it to accept a source directory of any python package in addition to sdists/bdist, which is already allowed.

As a bonus point, users could specify many pipeline packages as sources. The pipeline package is no different from any other packages being installed . In fact, some users build an sdist with their pipeline code and pass it via --extra_package.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

cc: @nitobuendia FYI