Storage: ConnectionError when uploading

[Indy2222]

Traceback:

  File \"/usr/local/lib/python2.7/dist-packages/google/cloud/storage/blob.py\", line 962, in upload_from_string
    content_type=content_type, client=client)
  File \"/usr/local/lib/python2.7/dist-packages/google/cloud/storage/blob.py\", line 886, in upload_from_file
    client, file_obj, content_type, size, num_retries)
  File \"/usr/local/lib/python2.7/dist-packages/google/cloud/storage/blob.py\", line 813, in _do_upload
    client, stream, content_type, size, num_retries)
  File \"/usr/local/lib/python2.7/dist-packages/google/cloud/storage/blob.py\", line 766, in _do_resumable_upload
    response = upload.transmit_next_chunk(transport)
  File \"/usr/local/lib/python2.7/dist-packages/google/resumable_media/requests/upload.py\", line 395, in transmit_next_chunk
    retry_strategy=self._retry_strategy)
  File \"/usr/local/lib/python2.7/dist-packages/google/resumable_media/requests/_helpers.py\", line 101, in http_request
    func, RequestsMixin._get_status_code, retry_strategy)
  File \"/usr/local/lib/python2.7/dist-packages/google/resumable_media/_helpers.py\", line 146, in wait_and_retry
    response = func()
  File \"/usr/local/lib/python2.7/dist-packages/google/auth/transport/requests.py\", line 179, in request
    method, url, data=data, headers=request_headers, **kwargs)
  File \"/usr/local/lib/python2.7/dist-packages/requests/sessions.py\", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File \"/usr/local/lib/python2.7/dist-packages/requests/sessions.py\", line 618, in send
    r = adapter.send(request, **kwargs)
  File \"/usr/local/lib/python2.7/dist-packages/requests/adapters.py\", line 490, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(\"(32, 'EPIPE')\",))

Setup:

$ python --version
Python 2.7.12

$ pip show show google-cloud
Name: google-cloud
Version: 0.27.0
Summary: API Client library for Google Cloud
Home-page: https://github.com/GoogleCloudPlatform/google-cloud-python
Author: Google Cloud Platform
Author-email: googleapis-publisher@google.com
License: Apache 2.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: google-cloud-bigquery, google-cloud-bigtable, google-cloud-core, google-cloud-datastore, google-cloud-dns, google-cloud-error-reporting, google-cloud-language, google-cloud-logging, google-cloud-monitoring, google-cloud-pubsub, google-cloud-resource-manager, google-cloud-runtimeconfig, google-cloud-spanner, google-cloud-speech, google-cloud-storage, google-cloud-translate, google-cloud-videointelligence, google-cloud-vision

$ pip freeze | grep requests
requests==2.18.4

$ pip freeze | grep urllib3
urllib3==1.22

[mehmetboraezer]

Hello, I am not sure it's a bug or not. Could you please share your code with us? By the way, I see your google-cloud version is old. Could you try to upgrade it?

[Indy2222]

The following is a simplified code which has failed:

client = storage.Client(project_name, scoped_credentials)
bucket = client.bucket(bucket_name)
blob = bucket.blob(path, 1048576)
blob.upload_from_string(content, content_type=content_type)

This issue is not easy to reproduce because it happens less than 1:1000 cases (rough estimate).

[dhermes]

@Indy2222 It's not unexpected that the low-level (i.e. socket-level) connection would fail very infrequently, and that's what this error indicates. That doesn't mean we can't catch this exception and retry, so thanks for reporting!

@jonparrott Do you know the retry policy that requests / urllib3 has?

[theacodes]

@jonparrott Do you know the retry policy that requests / urllib3 has?

It should retry connection errors a few times, but will give up pretty quickly. I'm not sure the exact semantics.

[dhermes]

@jonparrott Do you think we should try to catch requests.ConnectionError / requests.exceptions.ConnectionError?

[theacodes]

No. IMO, those are akin to socket-level errors.

[theacodes]

I am going to close this issue based on my last comment, but feel free to comment if this is still an issue.