Flaky System Tests

[daspecster]

@dhermes and I talked briefly about this issue. It seems like the tests are failing pretty consistently now.

What can we do to make this library or the tests more resilient?

Some of the issues seem to be account related?.

gcloud.exceptions.Forbidden: 403 Exceeded rate limits: Your dataset exceeded quota for dataset.insert or dataset.update per table. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors (POST https://www.googleapis.com/bigquery/v2/projects/precise-truck-742/datasets)

Maybe @jgeewax knows?

I think this is what @dhermes and I talked about last week. Is this where you were talking about adding/changing the retrying/exponential backoff?

[dhermes]

Ouch. I haven't seen a 403 like that before, isn't 429 supposed to be for rate limiting? Exponential backoff would hopefully resolve that.

The majority of the errors we've come across are due to eventual consistency though.

[daspecster]

@jgeewax I'm not sure how to move on this?

[dhermes]

I plan on tackling this soon.

The plan is to go through #1085, #1089, #1100 and others and take down the errors one by one. I developed a strategy in #535 and will try to resurrect it.

[daspecster]

Nice! Let me know if I can help in anyway.

I see in the docs for bigquery as well as for storage that the 403 response code is used for a lot of situations.
https://cloud.google.com/bigquery/troubleshooting-errors

[daspecster]

@jgeewax, do we need to up our quota limits? It looks like our normal number of builds is getting close to the limits?

======================================================================
ERROR: test_update_dataset (bigquery.TestBigQuery)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/bigquery.py", line 87, in test_update_dataset
    dataset.create()
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/gcloud/bigquery/dataset.py", line 423, in create
    method='POST', path=path, data=self._build_resource())
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/gcloud/connection.py", line 344, in api_request
    error_info=method + ' ' + url)
gcloud.exceptions.Forbidden: 403 Exceeded rate limits: Your dataset exceeded quota for dataset.insert or dataset.update per table. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors (POST https://www.googleapis.com/bigquery/v2/projects/precise-truck-742/datasets)
======================================================================
ERROR: test_update_table (bigquery.TestBigQuery)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/bigquery.py", line 191, in test_update_table
    dataset.create()
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/gcloud/bigquery/dataset.py", line 423, in create
    method='POST', path=path, data=self._build_resource())
  File "/home/travis/build/GoogleCloudPlatform/gcloud-python/gcloud/connection.py", line 344, in api_request
    error_info=method + ' ' + url)
gcloud.exceptions.Forbidden: 403 Exceeded rate limits: Your dataset exceeded quota for dataset.insert or dataset.update per table. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors (POST https://www.googleapis.com/bigquery/v2/projects/precise-truck-742/datasets)
----------------------------------------------------------------------
Ran 11 tests in 28.436s
FAILED (errors=2)
ERROR: InvocationError: '/home/travis/build/GoogleCloudPlatform/gcloud-python/.tox/system-tests/bin/python /home/travis/build/GoogleCloudPlatform/gcloud-python/system_tests/attempt_system_tests.py'
___________________________________ summary ____________________________________
ERROR:   system-tests: commands failed

[tseaver]

@daspecster Those BigQuery rate limiting issues are fixed via the backoff/retry stuff added in #2050.