Specify GCP compute project in BigQuery Pusher executor

[codesue]

Problem: We (Twitter) typically use two different GCP projects when working with BigQuery: one for compute (where the query is executed) and one for storage (where the table is located). With the current BigQuery Pusher executor, we have to use bigquery_serving_args[_PROJECT_ID_KEY] for both compute and storage, so our jobs fail due to permissions issues.

Proposed solution: Add support for specifying a compute project id that is different from the storage project id. It would be nice to have such a change backported to a 0.26.x version.

[casassg]

CC @rcrowe-google this would be important for us. Our set up has separate projects for query execution and model storage for BQ.

[casassg]

Re-upping this

[SinaChavoshi]

Hi Suzen,
Thank you so much for your contribution. Implementation looks great. I have one request, do you mind updating the documentation accordingly, see my comment below.

[SinaChavoshi]

Could you update the docstring (line 74 see below) to include a description of what this configuration is, what is the default behaviour and possibly an example of how this configuration can be used if the user has two projects per your usecase.

...
  def Do(self, input_dict: Dict[Text, List[types.Artifact]],
         output_dict: Dict[Text, List[types.Artifact]],
         exec_properties: Dict[Text, Any]):
    """Overrides the tfx_pusher_executor.
    Args:
    ...
      exec_properties: Mostly a passthrough input dict for
        tfx.components.Pusher.executor.  custom_config.bigquery_serving_args is
        consumed by this class.  For the full set of parameters supported by
        Big Query ML, refer to https://cloud.google.com/bigquery-ml/

[SinaChavoshi]

Could you also address the lint issues raised in presubmit

************* Module tfx.extensions.google_cloud_big_query.pusher.executor
tfx/extensions/google_cloud_big_query/pusher/executor.py:100:41: W1116: Second argument of isinstance is not a type (isinstance-second-argument-not-valid-type)
tfx/extensions/google_cloud_big_query/pusher/executor.py:147:6: W0707: Consider explicitly re-raising using the 'from' keyword (raise-missing-from)

************* Module tfx.extensions.google_cloud_big_query.pusher.executor_test
tfx/extensions/google_cloud_big_query/pusher/executor_test.py:38:4: R1725: Consider using Python 3 style super() without arguments (super-with-arguments)

[codesue]

The error Second argument of isinstance is not a type (isinstance-second-argument-not-valid-type) was coming from isinstance(custom_config, Dict). This was an issue in pylint: pylint-dev/pylint#3507.

[codesue]

@SinaChavoshi or @1025KB, may I have another review of this? 😄

[zhitaoli]

Could you also address the lint issues raised in presubmit

************* Module tfx.extensions.google_cloud_big_query.pusher.executor
tfx/extensions/google_cloud_big_query/pusher/executor.py:100:41: W1116: Second argument of isinstance is not a type (isinstance-second-argument-not-valid-type)
tfx/extensions/google_cloud_big_query/pusher/executor.py:147:6: W0707: Consider explicitly re-raising using the 'from' keyword (raise-missing-from)

************* Module tfx.extensions.google_cloud_big_query.pusher.executor_test
tfx/extensions/google_cloud_big_query/pusher/executor_test.py:38:4: R1725: Consider using Python 3 style super() without arguments (super-with-arguments)

Don't worry too much lint, we can address it while importing.