tests.system.test_blob: test_blob_update_storage_class_large_file failed

[flaky-bot]

This test failed!

To configure my behavior, see the Flaky Bot documentation.

If I'm commenting on this issue too often, add the flakybot: quiet label and
I will stop commenting.

---

commit: 1d384bf
buildURL: Build Status, Sponge
status: failed

Test output

self = 
file_obj = <_io.BufferedReader name='/tmpfs/src/github/python-storage/tests/data/five-point-one-mb-file.zip'>
rewind = False, size = 5253120, content_type = 'application/zip'
num_retries = None, client = None, predefined_acl = None
if_generation_match = None, if_generation_not_match = None
if_metageneration_match = None, if_metageneration_not_match = None, timeout = 60
checksum = None
retry = 
def upload_from_file(
    self,
    file_obj,
    rewind=False,
    size=None,
    content_type=None,
    num_retries=None,
    client=None,
    predefined_acl=None,
    if_generation_match=None,
    if_generation_not_match=None,
    if_metageneration_match=None,
    if_metageneration_not_match=None,
    timeout=_DEFAULT_TIMEOUT,
    checksum=None,
    retry=DEFAULT_RETRY_IF_GENERATION_SPECIFIED,
):
    """Upload the contents of this blob from a file-like object.

    The content type of the upload will be determined in order
    of precedence:

    - The value passed in to this method (if not :data:`None`)
    - The value stored on the current blob
    - The default value ('application/octet-stream')

    .. note::
       The effect of uploading to an existing blob depends on the
       "versioning" and "lifecycle" policies defined on the blob's
       bucket.  In the absence of those policies, upload will
       overwrite any existing contents.

       See the [`object versioning`](https://cloud.google.com/storage/docs/object-versioning)
       and [`lifecycle`](https://cloud.google.com/storage/docs/lifecycle)
       API documents for details.

    If the size of the data to be uploaded exceeds 8 MB a resumable media
    request will be used, otherwise the content and the metadata will be
    uploaded in a single multipart upload request.

    For more fine-grained over the upload process, check out
    [`google-resumable-media`](https://googleapis.dev/python/google-resumable-media/latest/index.html).

    If :attr:`user_project` is set on the bucket, bills the API request
    to that project.

    :type file_obj: file
    :param file_obj: A file handle opened in binary mode for reading.

    :type rewind: bool
    :param rewind:
        If True, seek to the beginning of the file handle before writing
        the file to Cloud Storage.

    :type size: int
    :param size:
        The number of bytes to be uploaded (which will be read from
        ``file_obj``). If not provided, the upload will be concluded once
        ``file_obj`` is exhausted.

    :type content_type: str
    :param content_type: (Optional) Type of content being uploaded.

    :type num_retries: int
    :param num_retries:
        Number of upload retries. By default, only uploads with
        if_generation_match set will be retried, as uploads without the
        argument are not guaranteed to be idempotent. Setting num_retries
        will override this default behavior and guarantee retries even when
        if_generation_match is not set.  (Deprecated: This argument
        will be removed in a future release.)

    :type client: :class:`~google.cloud.storage.client.Client`
    :param client:
        (Optional) The client to use.  If not passed, falls back to the
        ``client`` stored on the blob's bucket.

    :type predefined_acl: str
    :param predefined_acl: (Optional) Predefined access control list

    :type if_generation_match: long
    :param if_generation_match:
        (Optional) See :ref:`using-if-generation-match`

    :type if_generation_not_match: long
    :param if_generation_not_match:
        (Optional) See :ref:`using-if-generation-not-match`

    :type if_metageneration_match: long
    :param if_metageneration_match:
        (Optional) See :ref:`using-if-metageneration-match`

    :type if_metageneration_not_match: long
    :param if_metageneration_not_match:
        (Optional) See :ref:`using-if-metageneration-not-match`

    :type timeout: float or tuple
    :param timeout:
        (Optional) The amount of time, in seconds, to wait
        for the server response.  See: :ref:`configuring_timeouts`

    :type checksum: str
    :param checksum:
        (Optional) The type of checksum to compute to verify
        the integrity of the object. If the upload is completed in a single
        request, the checksum will be entirely precomputed and the remote
        server will handle verification and error handling. If the upload
        is too large and must be transmitted in multiple requests, the
        checksum will be incrementally computed and the client will handle
        verification and error handling, raising
        google.resumable_media.common.DataCorruption on a mismatch and
        attempting to delete the corrupted file. Supported values are
        "md5", "crc32c" and None. The default is None.

    :type retry: google.api_core.retry.Retry or google.cloud.storage.retry.ConditionalRetryPolicy
    :param retry: (Optional) How to retry the RPC. A None value will disable
        retries. A google.api_core.retry.Retry value will enable retries,
        and the object will define retriable response codes and errors and
        configure backoff and timeout options.

        A google.cloud.storage.retry.ConditionalRetryPolicy value wraps a
        Retry object and activates it only if certain conditions are met.
        This class exists to provide safe defaults for RPC calls that are
        not technically safe to retry normally (due to potential data
        duplication or other side-effects) but become safe to retry if a
        condition such as if_generation_match is set.

        See the retry.py source code and docstrings in this package
        (google.cloud.storage.retry) for information on retry types and how
        to configure them.

        Media operations (downloads and uploads) do not support non-default
        predicates in a Retry object. The default will always be used. Other
        configuration changes for Retry objects such as delays and deadlines
        are respected.

    :raises: :class:`~google.cloud.exceptions.GoogleCloudError`
             if the upload response returns an error status.
    """
    if num_retries is not None:
        warnings.warn(_NUM_RETRIES_MESSAGE, DeprecationWarning, stacklevel=2)
        # num_retries and retry are mutually exclusive. If num_retries is
        # set and retry is exactly the default, then nullify retry for
        # backwards compatibility.
        if retry is DEFAULT_RETRY_IF_GENERATION_SPECIFIED:
            retry = None

    _maybe_rewind(file_obj, rewind=rewind)
    predefined_acl = ACL.validate_predefined(predefined_acl)

    try:


      created_json = self._do_upload(


            client,
            file_obj,
            content_type,
            size,
            num_retries,
            predefined_acl,
            if_generation_match,
            if_generation_not_match,
            if_metageneration_match,
            if_metageneration_not_match,
            timeout=timeout,
            checksum=checksum,
            retry=retry,
        )

google/cloud/storage/blob.py:2539:

self = <Blob: gcp-systest-1673285927243, BigFilee8c88426ca414d60bc42bb0051427bc1, None>

client = None

stream = <_io.BufferedReader name='/tmpfs/src/github/python-storage/tests/data/five-point-one-mb-file.zip'>

content_type = 'application/zip', size = 5253120, num_retries = None

predefined_acl = None, if_generation_match = None

if_generation_not_match = None, if_metageneration_match = None

if_metageneration_not_match = None, timeout = 60, checksum = None, retry = None
def _do_upload(
    self,
    client,
    stream,
    content_type,
    size,
    num_retries,
    predefined_acl,
    if_generation_match,
    if_generation_not_match,
    if_metageneration_match,
    if_metageneration_not_match,
    timeout=_DEFAULT_TIMEOUT,
    checksum=None,
    retry=None,
):
    """Determine an upload strategy and then perform the upload.

    If the size of the data to be uploaded exceeds 8 MB a resumable media
    request will be used, otherwise the content and the metadata will be
    uploaded in a single multipart upload request.

    The content type of the upload will be determined in order
    of precedence:

    - The value passed in to this method (if not :data:`None`)
    - The value stored on the current blob
    - The default value ('application/octet-stream')

    :type client: :class:`~google.cloud.storage.client.Client`
    :param client:
        (Optional) The client to use.  If not passed, falls back to the
        ``client`` stored on the blob's bucket.

    :type stream: IO[bytes]
    :param stream: A bytes IO object open for reading.

    :type content_type: str
    :param content_type: Type of content being uploaded (or :data:`None`).

    :type size: int
    :param size:
        The number of bytes to be uploaded (which will be read from
        ``stream``). If not provided, the upload will be concluded once
        ``stream`` is exhausted (or :data:`None`).

    :type num_retries: int
    :param num_retries:
        Number of upload retries. By default, only uploads with
        if_generation_match set will be retried, as uploads without the
        argument are not guaranteed to be idempotent. Setting num_retries
        will override this default behavior and guarantee retries even when
        if_generation_match is not set.  (Deprecated: This argument
        will be removed in a future release.)

    :type predefined_acl: str
    :param predefined_acl: (Optional) Predefined access control list

    :type if_generation_match: long
    :param if_generation_match:
        (Optional) See :ref:`using-if-generation-match`

    :type if_generation_not_match: long
    :param if_generation_not_match:
        (Optional) See :ref:`using-if-generation-not-match`

    :type if_metageneration_match: long
    :param if_metageneration_match:
        (Optional) See :ref:`using-if-metageneration-match`

    :type if_metageneration_not_match: long
    :param if_metageneration_not_match:
        (Optional) See :ref:`using-if-metageneration-not-match`

    :type timeout: float or tuple
    :param timeout:
        (Optional) The amount of time, in seconds, to wait
        for the server response.  See: :ref:`configuring_timeouts`

    :type checksum: str
    :param checksum:
        (Optional) The type of checksum to compute to verify
        the integrity of the object. If the upload is completed in a single
        request, the checksum will be entirely precomputed and the remote
        server will handle verification and error handling. If the upload
        is too large and must be transmitted in multiple requests, the
        checksum will be incrementally computed and the client will handle
        verification and error handling, raising
        google.resumable_media.common.DataCorruption on a mismatch and
        attempting to delete the corrupted file. Supported values are
        "md5", "crc32c" and None. The default is None.

    :type retry: google.api_core.retry.Retry or google.cloud.storage.retry.ConditionalRetryPolicy
    :param retry: (Optional) How to retry the RPC. A None value will disable
        retries. A google.api_core.retry.Retry value will enable retries,
        and the object will define retriable response codes and errors and
        configure backoff and timeout options.

        A google.cloud.storage.retry.ConditionalRetryPolicy value wraps a
        Retry object and activates it only if certain conditions are met.
        This class exists to provide safe defaults for RPC calls that are
        not technically safe to retry normally (due to potential data
        duplication or other side-effects) but become safe to retry if a
        condition such as if_generation_match is set.

        See the retry.py source code and docstrings in this package
        (google.cloud.storage.retry) for information on retry types and how
        to configure them.

        Media operations (downloads and uploads) do not support non-default
        predicates in a Retry object. The default will always be used. Other
        configuration changes for Retry objects such as delays and deadlines
        are respected.

    :rtype: dict
    :returns: The parsed JSON from the "200 OK" response. This will be the
              **only** response in the multipart case and it will be the
              **final** response in the resumable case.
    """

    # Handle ConditionalRetryPolicy.
    if isinstance(retry, ConditionalRetryPolicy):
        # Conditional retries are designed for non-media calls, which change
        # arguments into query_params dictionaries. Media operations work
        # differently, so here we make a "fake" query_params to feed to the
        # ConditionalRetryPolicy.
        query_params = {
            "ifGenerationMatch": if_generation_match,
            "ifMetagenerationMatch": if_metageneration_match,
        }
        retry = retry.get_retry_policy_if_conditions_met(query_params=query_params)

    if size is not None and size <= _MAX_MULTIPART_SIZE:


      response = self._do_multipart_upload(


            client,
            stream,
            content_type,
            size,
            num_retries,
            predefined_acl,
            if_generation_match,
            if_generation_not_match,
            if_metageneration_match,
            if_metageneration_not_match,
            timeout=timeout,
            checksum=checksum,
            retry=retry,
        )

google/cloud/storage/blob.py:2354:

self = <Blob: gcp-systest-1673285927243, BigFilee8c88426ca414d60bc42bb0051427bc1, None>

client = <google.cloud.storage.client.Client object at 0x7ffa6cb81070>

stream = <_io.BufferedReader name='/tmpfs/src/github/python-storage/tests/data/five-point-one-mb-file.zip'>

content_type = 'application/zip', size = 5253120, num_retries = None

predefined_acl = None, if_generation_match = None

if_generation_not_match = None, if_metageneration_match = None

if_metageneration_not_match = None, timeout = 60, checksum = None, retry = None
def _do_multipart_upload(
    self,
    client,
    stream,
    content_type,
    size,
    num_retries,
    predefined_acl,
    if_generation_match,
    if_generation_not_match,
    if_metageneration_match,
    if_metageneration_not_match,
    timeout=_DEFAULT_TIMEOUT,
    checksum=None,
    retry=None,
):
    """Perform a multipart upload.

    The content type of the upload will be determined in order
    of precedence:

    - The value passed in to this method (if not :data:`None`)
    - The value stored on the current blob
    - The default value ('application/octet-stream')

    :type client: :class:`~google.cloud.storage.client.Client`
    :param client:
        (Optional) The client to use.  If not passed, falls back to the
        ``client`` stored on the blob's bucket.

    :type stream: IO[bytes]
    :param stream: A bytes IO object open for reading.

    :type content_type: str
    :param content_type: Type of content being uploaded (or :data:`None`).

    :type size: int
    :param size:
        The number of bytes to be uploaded (which will be read from
        ``stream``). If not provided, the upload will be concluded once
        ``stream`` is exhausted (or :data:`None`).

    :type num_retries: int
    :param num_retries:
        Number of upload retries. By default, only uploads with
        if_generation_match set will be retried, as uploads without the
        argument are not guaranteed to be idempotent. Setting num_retries
        will override this default behavior and guarantee retries even when
        if_generation_match is not set.  (Deprecated: This argument
        will be removed in a future release.)

    :type predefined_acl: str
    :param predefined_acl: (Optional) Predefined access control list

    :type if_generation_match: long
    :param if_generation_match:
        (Optional) See :ref:`using-if-generation-match`

    :type if_generation_not_match: long
    :param if_generation_not_match:
        (Optional) See :ref:`using-if-generation-not-match`

    :type if_metageneration_match: long
    :param if_metageneration_match:
        (Optional) See :ref:`using-if-metageneration-match`

    :type if_metageneration_not_match: long
    :param if_metageneration_not_match:
        (Optional) See :ref:`using-if-metageneration-not-match`

    :type timeout: float or tuple
    :param timeout:
        (Optional) The amount of time, in seconds, to wait
        for the server response.  See: :ref:`configuring_timeouts`

    :type checksum: str
    :param checksum:
        (Optional) The type of checksum to compute to verify
        the integrity of the object. The request metadata will be amended
        to include the computed value. Using this option will override a
        manually-set checksum value. Supported values are "md5",
        "crc32c" and None. The default is None.

    :type retry: google.api_core.retry.Retry
    :param retry: (Optional) How to retry the RPC. A None value will disable
        retries. A google.api_core.retry.Retry value will enable retries,
        and the object will configure backoff and timeout options. Custom
        predicates (customizable error codes) are not supported for media
        operations such as this one.

        This private method does not accept ConditionalRetryPolicy values
        because the information necessary to evaluate the policy is instead
        evaluated in blob._do_upload().

        See the retry.py source code and docstrings in this package
        (google.cloud.storage.retry) for information on retry types and how
        to configure them.

    :rtype: :class:`~requests.Response`
    :returns: The "200 OK" response object returned after the multipart
              upload request.
    :raises: :exc:`ValueError` if ``size`` is not :data:`None` but the
             ``stream`` has fewer than ``size`` bytes remaining.
    """
    if size is None:
        data = stream.read()
    else:
        data = stream.read(size)
        if len(data) < size:
            msg = _READ_LESS_THAN_SIZE.format(size, len(data))
            raise ValueError(msg)

    client = self._require_client(client)
    transport = self._get_transport(client)
    if "metadata" in self._properties and "metadata" not in self._changes:
        self._changes.add("metadata")
    info = self._get_upload_arguments(client, content_type)
    headers, object_metadata, content_type = info

    hostname = _get_host_name(client._connection)
    base_url = _MULTIPART_URL_TEMPLATE.format(
        hostname=hostname, bucket_path=self.bucket.path, api_version=_API_VERSION
    )
    name_value_pairs = []

    if self.user_project is not None:
        name_value_pairs.append(("userProject", self.user_project))

    # When a Customer Managed Encryption Key is used to encrypt Cloud Storage object
    # at rest, object resource metadata will store the version of the Key Management
    # Service cryptographic material. If a Blob instance with KMS Key metadata set is
    # used to upload a new version of the object then the existing kmsKeyName version
    # value can't be used in the upload request and the client instead ignores it.
    if (
        self.kms_key_name is not None
        and "cryptoKeyVersions" not in self.kms_key_name
    ):
        name_value_pairs.append(("kmsKeyName", self.kms_key_name))

    if predefined_acl is not None:
        name_value_pairs.append(("predefinedAcl", predefined_acl))

    if if_generation_match is not None:
        name_value_pairs.append(("ifGenerationMatch", if_generation_match))

    if if_generation_not_match is not None:
        name_value_pairs.append(("ifGenerationNotMatch", if_generation_not_match))

    if if_metageneration_match is not None:
        name_value_pairs.append(("ifMetagenerationMatch", if_metageneration_match))

    if if_metageneration_not_match is not None:
        name_value_pairs.append(
            ("ifMetaGenerationNotMatch", if_metageneration_not_match)
        )

    upload_url = _add_query_parameters(base_url, name_value_pairs)
    upload = MultipartUpload(upload_url, headers=headers, checksum=checksum)

    upload._retry_strategy = _api_core_retry_to_resumable_media_retry(
        retry, num_retries
    )


  response = upload.transmit(


        transport, data, object_metadata, content_type, timeout=timeout
    )

google/cloud/storage/blob.py:1889:

self = <google.resumable_media.requests.upload.MultipartUpload object at 0x7ffa6cdfa0d0>

transport = <google.auth.transport.requests.AuthorizedSession object at 0x7ffa6ccc82b0>

data = b'y\x99\xf5\t\xc5\xc2\xc8N6\xf7D.\xce\xeey-\x14IU\xcdPf\xdd\x85\x8cV\xd3.\xeb(\xd8ULF|O\x97\xa7-\xa8!\xb4&\x9e\xe8Q\xe...82\xf2\x9f\xf7\x965\xd7w;.\x94L\x8a\xeb!w8U\xef\x9a+\xbe\x1d\x0b\x98\x94\x8d\xdc\xcd{k\x08\x93\xd4\xe9\x06\xd1\xb9\x7f'

metadata = {'name': 'BigFilee8c88426ca414d60bc42bb0051427bc1'}

content_type = 'application/zip', timeout = 60
def transmit(
    self,
    transport,
    data,
    metadata,
    content_type,
    timeout=(
        _request_helpers._DEFAULT_CONNECT_TIMEOUT,
        _request_helpers._DEFAULT_READ_TIMEOUT,
    ),
):
    """Transmit the resource to be uploaded.

    Args:
        transport (~requests.Session): A ``requests`` object which can
            make authenticated requests.
        data (bytes): The resource content to be uploaded.
        metadata (Mapping[str, str]): The resource metadata, such as an
            ACL list.
        content_type (str): The content type of the resource, e.g. a JPEG
            image has content type ``image/jpeg``.
        timeout (Optional[Union[float, Tuple[float, float]]]):
            The number of seconds to wait for the server response.
            Depending on the retry strategy, a request may be repeated
            several times using the same timeout each time.

            Can also be passed as a tuple (connect_timeout, read_timeout).
            See :meth:`requests.Session.request` documentation for details.

    Returns:
        ~requests.Response: The HTTP response returned by ``transport``.
    """
    method, url, payload, headers = self._prepare_request(
        data, metadata, content_type
    )

    # Wrap the request business logic in a function to be retried.
    def retriable_request():
        result = transport.request(
            method, url, data=payload, headers=headers, timeout=timeout
        )

        self._process_response(result)

        return result


  return _request_helpers.wait_and_retry(


        retriable_request, self._get_status_code, self._retry_strategy
    )

.nox/system-3-8/lib/python3.8/site-packages/google/resumable_media/requests/upload.py:153:

func = <function MultipartUpload.transmit..retriable_request at 0x7ffa6af49040>

get_status_code = <function RequestsMixin._get_status_code at 0x7ffa6d0f24c0>

retry_strategy = <google.resumable_media.common.RetryStrategy object at 0x7ffa6cdfa2b0>
def wait_and_retry(func, get_status_code, retry_strategy):
    """Attempts to retry a call to ``func`` until success.

    Expects ``func`` to return an HTTP response and uses ``get_status_code``
    to check if the response is retry-able.

    ``func`` is expected to raise a failure status code as a
    common.InvalidResponse, at which point this method will check the code
    against the common.RETRIABLE list of retriable status codes.

    Will retry until :meth:`~.RetryStrategy.retry_allowed` (on the current
    ``retry_strategy``) returns :data:`False`. Uses
    :func:`_helpers.calculate_retry_wait` to double the wait time (with jitter)
    after each attempt.

    Args:
        func (Callable): A callable that takes no arguments and produces
            an HTTP response which will be checked as retry-able.
        get_status_code (Callable[Any, int]): Helper to get a status code
            from a response.
        retry_strategy (~google.resumable_media.common.RetryStrategy): The
            strategy to use if the request fails and must be retried.

    Returns:
        object: The return value of ``func``.
    """
    total_sleep = 0.0
    num_retries = 0
    # base_wait will be multiplied by the multiplier on the first retry.
    base_wait = float(retry_strategy.initial_delay) / retry_strategy.multiplier

    # Set the retriable_exception_type if possible. We expect requests to be
    # present here and the transport to be using requests.exceptions errors,
    # but due to loose coupling with the transport layer we can't guarantee it.

    while True:  # return on success or when retries exhausted.
        error = None
        try:
            response = func()
        except _CONNECTION_ERROR_CLASSES as e:
            error = e  # Fall through to retry, if there are retries left.
        except common.InvalidResponse as e:
            # An InvalidResponse is only retriable if its status code matches.
            # The `process_response()` method on a Download or Upload method
            # will convert the status code into an exception.
            if get_status_code(e.response) in common.RETRYABLE:
                error = e  # Fall through to retry, if there are retries left.
            else:
                raise  # If the status code is not retriable, raise w/o retry.
        else:
            return response

        base_wait, wait_time = _helpers.calculate_retry_wait(
            base_wait, retry_strategy.max_sleep, retry_strategy.multiplier
        )
        num_retries += 1
        total_sleep += wait_time

        # Check if (another) retry is allowed. If retries are exhausted and
        # no acceptable response was received, raise the retriable error.
        if not retry_strategy.retry_allowed(total_sleep, num_retries):


          raise error


.nox/system-3-8/lib/python3.8/site-packages/google/resumable_media/requests/_request_helpers.py:171:

func = <function MultipartUpload.transmit..retriable_request at 0x7ffa6af49040>

get_status_code = <function RequestsMixin._get_status_code at 0x7ffa6d0f24c0>

retry_strategy = <google.resumable_media.common.RetryStrategy object at 0x7ffa6cdfa2b0>
def wait_and_retry(func, get_status_code, retry_strategy):
    """Attempts to retry a call to ``func`` until success.

    Expects ``func`` to return an HTTP response and uses ``get_status_code``
    to check if the response is retry-able.

    ``func`` is expected to raise a failure status code as a
    common.InvalidResponse, at which point this method will check the code
    against the common.RETRIABLE list of retriable status codes.

    Will retry until :meth:`~.RetryStrategy.retry_allowed` (on the current
    ``retry_strategy``) returns :data:`False`. Uses
    :func:`_helpers.calculate_retry_wait` to double the wait time (with jitter)
    after each attempt.

    Args:
        func (Callable): A callable that takes no arguments and produces
            an HTTP response which will be checked as retry-able.
        get_status_code (Callable[Any, int]): Helper to get a status code
            from a response.
        retry_strategy (~google.resumable_media.common.RetryStrategy): The
            strategy to use if the request fails and must be retried.

    Returns:
        object: The return value of ``func``.
    """
    total_sleep = 0.0
    num_retries = 0
    # base_wait will be multiplied by the multiplier on the first retry.
    base_wait = float(retry_strategy.initial_delay) / retry_strategy.multiplier

    # Set the retriable_exception_type if possible. We expect requests to be
    # present here and the transport to be using requests.exceptions errors,
    # but due to loose coupling with the transport layer we can't guarantee it.

    while True:  # return on success or when retries exhausted.
        error = None
        try:


          response = func()


.nox/system-3-8/lib/python3.8/site-packages/google/resumable_media/requests/_request_helpers.py:148:

def retriable_request():
    result = transport.request(
        method, url, data=payload, headers=headers, timeout=timeout
    )


  self._process_response(result)


.nox/system-3-8/lib/python3.8/site-packages/google/resumable_media/requests/upload.py:149:

self = <google.resumable_media.requests.upload.MultipartUpload object at 0x7ffa6cdfa0d0>

response = <Response [503]>
def _process_response(self, response):
    """Process the response from an HTTP request.

    This is everything that must be done after a request that doesn't
    require network I/O (or other I/O). This is based on the `sans-I/O`_
    philosophy.

    Args:
        response (object): The HTTP response object.

    Raises:
        ~google.resumable_media.common.InvalidResponse: If the status
            code is not 200.

    .. _sans-I/O: https://sans-io.readthedocs.io/
    """
    # Tombstone the current upload so it cannot be used again (in either
    # failure or success).
    self._finished = True


  _helpers.require_status_code(response, (http.client.OK,), self._get_status_code)


.nox/system-3-8/lib/python3.8/site-packages/google/resumable_media/_upload.py:114:

response = <Response [503]>, status_codes = (<HTTPStatus.OK: 200>,)

get_status_code = <function RequestsMixin._get_status_code at 0x7ffa6d0f24c0>

callback = <function do_nothing at 0x7ffa6d0d0f70>
def require_status_code(response, status_codes, get_status_code, callback=do_nothing):
    """Require a response has a status code among a list.

    Args:
        response (object): The HTTP response object.
        status_codes (tuple): The acceptable status codes.
        get_status_code (Callable[Any, int]): Helper to get a status code
            from a response.
        callback (Optional[Callable]): A callback that takes no arguments,
            to be executed when an exception is being raised.

    Returns:
        int: The status code.

    Raises:
        ~google.resumable_media.common.InvalidResponse: If the status code
            is not one of the values in ``status_codes``.
    """
    status_code = get_status_code(response)
    if status_code not in status_codes:
        if status_code not in common.RETRYABLE:
            callback()


      raise common.InvalidResponse(


            response,
            "Request failed with status code",
            status_code,
            "Expected one of",
            *status_codes
        )

E           google.resumable_media.common.InvalidResponse: ('Request failed with status code', 503, 'Expected one of', <HTTPStatus.OK: 200>)
.nox/system-3-8/lib/python3.8/site-packages/google/resumable_media/_helpers.py:108: InvalidResponse
During handling of the above exception, another exception occurred:
shared_bucket = <Bucket: gcp-systest-1673285927243>, blobs_to_delete = []

file_data = {'big': {'hash': b'cEome4a+NYd7YIXzXQnR5Q==', 'path': '/tmpfs/src/github/python-storage/tests/data/five-point-one-mb-f...g'}, 'simple': {'hash': b'3Hkwjv2WvCnKjNR6Z3CboA==', 'path': '/tmpfs/src/github/python-storage/tests/data/simple.txt'}}
def test_blob_update_storage_class_large_file(
    shared_bucket, blobs_to_delete, file_data
):
    from google.cloud.storage import constants

    blob = shared_bucket.blob(f"BigFile{uuid.uuid4().hex}")

    info = file_data["big"]


  blob.upload_from_filename(info["path"])


tests/system/test_blob.py:1113:

google/cloud/storage/blob.py:2683: in upload_from_filename

self.upload_from_file(

google/cloud/storage/blob.py:2556: in upload_from_file

_raise_from_invalid_response(exc)

error = InvalidResponse('Request failed with status code', 503, 'Expected one of', <HTTPStatus.OK: 200>)
def _raise_from_invalid_response(error):
    """Re-wrap and raise an ``InvalidResponse`` exception.

    :type error: :exc:`google.resumable_media.InvalidResponse`
    :param error: A caught exception from the ``google-resumable-media``
                  library.

    :raises: :class:`~google.cloud.exceptions.GoogleCloudError` corresponding
             to the failed status code
    """
    response = error.response

    # The 'response.text' gives the actual reason of error, where 'error' gives
    # the message of expected status code.
    if response.text:
        error_message = response.text + ": " + str(error)
    else:
        error_message = str(error)

    message = f"{response.request.method} {response.request.url}: {error_message}"


  raise exceptions.from_http_status(response.status_code, message, response=response)


E       google.api_core.exceptions.ServiceUnavailable: 503 POST https://storage-preprod-test-unified.googleusercontent.com/upload/storage/v1_preprod/b/gcp-systest-1673285927243/o?uploadType=multipart: Service Unavailable: ('Request failed with status code', 503, 'Expected one of', <HTTPStatus.OK: 200>)
google/cloud/storage/blob.py:4368: ServiceUnavailable

[flaky-bot]

Test passed for commit ba323d8 (Build Status, Sponge)! Closing this issue.

[flaky-bot]

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

commit: 1d384bf
buildURL: Build Status, Sponge
status: failed

Test output

self = 
file_obj = <_io.BufferedReader name='/tmpfs/src/github/python-storage/tests/data/five-point-one-mb-file.zip'>
rewind = False, size = 5253120, content_type = 'application/zip'
num_retries = None, client = None, predefined_acl = None
if_generation_match = None, if_generation_not_match = None
if_metageneration_match = None, if_metageneration_not_match = None, timeout = 60
checksum = None
retry = 
def upload_from_file(
    self,
    file_obj,
    rewind=False,
    size=None,
    content_type=None,
    num_retries=None,
    client=None,
    predefined_acl=None,
    if_generation_match=None,
    if_generation_not_match=None,
    if_metageneration_match=None,
    if_metageneration_not_match=None,
    timeout=_DEFAULT_TIMEOUT,
    checksum=None,
    retry=DEFAULT_RETRY_IF_GENERATION_SPECIFIED,
):
    """Upload the contents of this blob from a file-like object.

    The content type of the upload will be determined in order
    of precedence:

    - The value passed in to this method (if not :data:`None`)
    - The value stored on the current blob
    - The default value ('application/octet-stream')

    .. note::
       The effect of uploading to an existing blob depends on the
       "versioning" and "lifecycle" policies defined on the blob's
       bucket.  In the absence of those policies, upload will
       overwrite any existing contents.

       See the [`object versioning`](https://cloud.google.com/storage/docs/object-versioning)
       and [`lifecycle`](https://cloud.google.com/storage/docs/lifecycle)
       API documents for details.

    If the size of the data to be uploaded exceeds 8 MB a resumable media
    request will be used, otherwise the content and the metadata will be
    uploaded in a single multipart upload request.

    For more fine-grained over the upload process, check out
    [`google-resumable-media`](https://googleapis.dev/python/google-resumable-media/latest/index.html).

    If :attr:`user_project` is set on the bucket, bills the API request
    to that project.

    :type file_obj: file
    :param file_obj: A file handle opened in binary mode for reading.

    :type rewind: bool
    :param rewind:
        If True, seek to the beginning of the file handle before writing
        the file to Cloud Storage.

    :type size: int
    :param size:
        The number of bytes to be uploaded (which will be read from
        ``file_obj``). If not provided, the upload will be concluded once
        ``file_obj`` is exhausted.

    :type content_type: str
    :param content_type: (Optional) Type of content being uploaded.

    :type num_retries: int
    :param num_retries:
        Number of upload retries. By default, only uploads with
        if_generation_match set will be retried, as uploads without the
        argument are not guaranteed to be idempotent. Setting num_retries
        will override this default behavior and guarantee retries even when
        if_generation_match is not set.  (Deprecated: This argument
        will be removed in a future release.)

    :type client: :class:`~google.cloud.storage.client.Client`
    :param client:
        (Optional) The client to use.  If not passed, falls back to the
        ``client`` stored on the blob's bucket.

    :type predefined_acl: str
    :param predefined_acl: (Optional) Predefined access control list

    :type if_generation_match: long
    :param if_generation_match:
        (Optional) See :ref:`using-if-generation-match`

    :type if_generation_not_match: long
    :param if_generation_not_match:
        (Optional) See :ref:`using-if-generation-not-match`

    :type if_metageneration_match: long
    :param if_metageneration_match:
        (Optional) See :ref:`using-if-metageneration-match`

    :type if_metageneration_not_match: long
    :param if_metageneration_not_match:
        (Optional) See :ref:`using-if-metageneration-not-match`

    :type timeout: float or tuple
    :param timeout:
        (Optional) The amount of time, in seconds, to wait
        for the server response.  See: :ref:`configuring_timeouts`

    :type checksum: str
    :param checksum:
        (Optional) The type of checksum to compute to verify
        the integrity of the object. If the upload is completed in a single
        request, the checksum will be entirely precomputed and the remote
        server will handle verification and error handling. If the upload
        is too large and must be transmitted in multiple requests, the
        checksum will be incrementally computed and the client will handle
        verification and error handling, raising
        google.resumable_media.common.DataCorruption on a mismatch and
        attempting to delete the corrupted file. Supported values are
        "md5", "crc32c" and None. The default is None.

    :type retry: google.api_core.retry.Retry or google.cloud.storage.retry.ConditionalRetryPolicy
    :param retry: (Optional) How to retry the RPC. A None value will disable
        retries. A google.api_core.retry.Retry value will enable retries,
        and the object will define retriable response codes and errors and
        configure backoff and timeout options.

        A google.cloud.storage.retry.ConditionalRetryPolicy value wraps a
        Retry object and activates it only if certain conditions are met.
        This class exists to provide safe defaults for RPC calls that are
        not technically safe to retry normally (due to potential data
        duplication or other side-effects) but become safe to retry if a
        condition such as if_generation_match is set.

        See the retry.py source code and docstrings in this package
        (google.cloud.storage.retry) for information on retry types and how
        to configure them.

        Media operations (downloads and uploads) do not support non-default
        predicates in a Retry object. The default will always be used. Other
        configuration changes for Retry objects such as delays and deadlines
        are respected.

    :raises: :class:`~google.cloud.exceptions.GoogleCloudError`
             if the upload response returns an error status.
    """
    if num_retries is not None:
        warnings.warn(_NUM_RETRIES_MESSAGE, DeprecationWarning, stacklevel=2)
        # num_retries and retry are mutually exclusive. If num_retries is
        # set and retry is exactly the default, then nullify retry for
        # backwards compatibility.
        if retry is DEFAULT_RETRY_IF_GENERATION_SPECIFIED:
            retry = None

    _maybe_rewind(file_obj, rewind=rewind)
    predefined_acl = ACL.validate_predefined(predefined_acl)

    try:


      created_json = self._do_upload(


            client,
            file_obj,
            content_type,
            size,
            num_retries,
            predefined_acl,
            if_generation_match,
            if_generation_not_match,
            if_metageneration_match,
            if_metageneration_not_match,
            timeout=timeout,
            checksum=checksum,
            retry=retry,
        )

google/cloud/storage/blob.py:2539:

self = <Blob: gcp-systest-1673330037529, BigFile07c10a4e8951433bb7bb8658d5c83c62, None>

client = None

stream = <_io.BufferedReader name='/tmpfs/src/github/python-storage/tests/data/five-point-one-mb-file.zip'>

content_type = 'application/zip', size = 5253120, num_retries = None

predefined_acl = None, if_generation_match = None

if_generation_not_match = None, if_metageneration_match = None

if_metageneration_not_match = None, timeout = 60, checksum = None, retry = None
def _do_upload(
    self,
    client,
    stream,
    content_type,
    size,
    num_retries,
    predefined_acl,
    if_generation_match,
    if_generation_not_match,
    if_metageneration_match,
    if_metageneration_not_match,
    timeout=_DEFAULT_TIMEOUT,
    checksum=None,
    retry=None,
):
    """Determine an upload strategy and then perform the upload.

    If the size of the data to be uploaded exceeds 8 MB a resumable media
    request will be used, otherwise the content and the metadata will be
    uploaded in a single multipart upload request.

    The content type of the upload will be determined in order
    of precedence:

    - The value passed in to this method (if not :data:`None`)
    - The value stored on the current blob
    - The default value ('application/octet-stream')

    :type client: :class:`~google.cloud.storage.client.Client`
    :param client:
        (Optional) The client to use.  If not passed, falls back to the
        ``client`` stored on the blob's bucket.

    :type stream: IO[bytes]
    :param stream: A bytes IO object open for reading.

    :type content_type: str
    :param content_type: Type of content being uploaded (or :data:`None`).

    :type size: int
    :param size:
        The number of bytes to be uploaded (which will be read from
        ``stream``). If not provided, the upload will be concluded once
        ``stream`` is exhausted (or :data:`None`).

    :type num_retries: int
    :param num_retries:
        Number of upload retries. By default, only uploads with
        if_generation_match set will be retried, as uploads without the
        argument are not guaranteed to be idempotent. Setting num_retries
        will override this default behavior and guarantee retries even when
        if_generation_match is not set.  (Deprecated: This argument
        will be removed in a future release.)

    :type predefined_acl: str
    :param predefined_acl: (Optional) Predefined access control list

    :type if_generation_match: long
    :param if_generation_match:
        (Optional) See :ref:`using-if-generation-match`

    :type if_generation_not_match: long
    :param if_generation_not_match:
        (Optional) See :ref:`using-if-generation-not-match`

    :type if_metageneration_match: long
    :param if_metageneration_match:
        (Optional) See :ref:`using-if-metageneration-match`

    :type if_metageneration_not_match: long
    :param if_metageneration_not_match:
        (Optional) See :ref:`using-if-metageneration-not-match`

    :type timeout: float or tuple
    :param timeout:
        (Optional) The amount of time, in seconds, to wait
        for the server response.  See: :ref:`configuring_timeouts`

    :type checksum: str
    :param checksum:
        (Optional) The type of checksum to compute to verify
        the integrity of the object. If the upload is completed in a single
        request, the checksum will be entirely precomputed and the remote
        server will handle verification and error handling. If the upload
        is too large and must be transmitted in multiple requests, the
        checksum will be incrementally computed and the client will handle
        verification and error handling, raising
        google.resumable_media.common.DataCorruption on a mismatch and
        attempting to delete the corrupted file. Supported values are
        "md5", "crc32c" and None. The default is None.

    :type retry: google.api_core.retry.Retry or google.cloud.storage.retry.ConditionalRetryPolicy
    :param retry: (Optional) How to retry the RPC. A None value will disable
        retries. A google.api_core.retry.Retry value will enable retries,
        and the object will define retriable response codes and errors and
        configure backoff and timeout options.

        A google.cloud.storage.retry.ConditionalRetryPolicy value wraps a
        Retry object and activates it only if certain conditions are met.
        This class exists to provide safe defaults for RPC calls that are
        not technically safe to retry normally (due to potential data
        duplication or other side-effects) but become safe to retry if a
        condition such as if_generation_match is set.

        See the retry.py source code and docstrings in this package
        (google.cloud.storage.retry) for information on retry types and how
        to configure them.

        Media operations (downloads and uploads) do not support non-default
        predicates in a Retry object. The default will always be used. Other
        configuration changes for Retry objects such as delays and deadlines
        are respected.

    :rtype: dict
    :returns: The parsed JSON from the "200 OK" response. This will be the
              **only** response in the multipart case and it will be the
              **final** response in the resumable case.
    """

    # Handle ConditionalRetryPolicy.
    if isinstance(retry, ConditionalRetryPolicy):
        # Conditional retries are designed for non-media calls, which change
        # arguments into query_params dictionaries. Media operations work
        # differently, so here we make a "fake" query_params to feed to the
        # ConditionalRetryPolicy.
        query_params = {
            "ifGenerationMatch": if_generation_match,
            "ifMetagenerationMatch": if_metageneration_match,
        }
        retry = retry.get_retry_policy_if_conditions_met(query_params=query_params)

    if size is not None and size <= _MAX_MULTIPART_SIZE:


      response = self._do_multipart_upload(


            client,
            stream,
            content_type,
            size,
            num_retries,
            predefined_acl,
            if_generation_match,
            if_generation_not_match,
            if_metageneration_match,
            if_metageneration_not_match,
            timeout=timeout,
            checksum=checksum,
            retry=retry,
        )

google/cloud/storage/blob.py:2354:

self = <Blob: gcp-systest-1673330037529, BigFile07c10a4e8951433bb7bb8658d5c83c62, None>

client = <google.cloud.storage.client.Client object at 0x7f73d8240b20>

stream = <_io.BufferedReader name='/tmpfs/src/github/python-storage/tests/data/five-point-one-mb-file.zip'>

content_type = 'application/zip', size = 5253120, num_retries = None

predefined_acl = None, if_generation_match = None

if_generation_not_match = None, if_metageneration_match = None

if_metageneration_not_match = None, timeout = 60, checksum = None, retry = None
def _do_multipart_upload(
    self,
    client,
    stream,
    content_type,
    size,
    num_retries,
    predefined_acl,
    if_generation_match,
    if_generation_not_match,
    if_metageneration_match,
    if_metageneration_not_match,
    timeout=_DEFAULT_TIMEOUT,
    checksum=None,
    retry=None,
):
    """Perform a multipart upload.

    The content type of the upload will be determined in order
    of precedence:

    - The value passed in to this method (if not :data:`None`)
    - The value stored on the current blob
    - The default value ('application/octet-stream')

    :type client: :class:`~google.cloud.storage.client.Client`
    :param client:
        (Optional) The client to use.  If not passed, falls back to the
        ``client`` stored on the blob's bucket.

    :type stream: IO[bytes]
    :param stream: A bytes IO object open for reading.

    :type content_type: str
    :param content_type: Type of content being uploaded (or :data:`None`).

    :type size: int
    :param size:
        The number of bytes to be uploaded (which will be read from
        ``stream``). If not provided, the upload will be concluded once
        ``stream`` is exhausted (or :data:`None`).

    :type num_retries: int
    :param num_retries:
        Number of upload retries. By default, only uploads with
        if_generation_match set will be retried, as uploads without the
        argument are not guaranteed to be idempotent. Setting num_retries
        will override this default behavior and guarantee retries even when
        if_generation_match is not set.  (Deprecated: This argument
        will be removed in a future release.)

    :type predefined_acl: str
    :param predefined_acl: (Optional) Predefined access control list

    :type if_generation_match: long
    :param if_generation_match:
        (Optional) See :ref:`using-if-generation-match`

    :type if_generation_not_match: long
    :param if_generation_not_match:
        (Optional) See :ref:`using-if-generation-not-match`

    :type if_metageneration_match: long
    :param if_metageneration_match:
        (Optional) See :ref:`using-if-metageneration-match`

    :type if_metageneration_not_match: long
    :param if_metageneration_not_match:
        (Optional) See :ref:`using-if-metageneration-not-match`

    :type timeout: float or tuple
    :param timeout:
        (Optional) The amount of time, in seconds, to wait
        for the server response.  See: :ref:`configuring_timeouts`

    :type checksum: str
    :param checksum:
        (Optional) The type of checksum to compute to verify
        the integrity of the object. The request metadata will be amended
        to include the computed value. Using this option will override a
        manually-set checksum value. Supported values are "md5",
        "crc32c" and None. The default is None.

    :type retry: google.api_core.retry.Retry
    :param retry: (Optional) How to retry the RPC. A None value will disable
        retries. A google.api_core.retry.Retry value will enable retries,
        and the object will configure backoff and timeout options. Custom
        predicates (customizable error codes) are not supported for media
        operations such as this one.

        This private method does not accept ConditionalRetryPolicy values
        because the information necessary to evaluate the policy is instead
        evaluated in blob._do_upload().

        See the retry.py source code and docstrings in this package
        (google.cloud.storage.retry) for information on retry types and how
        to configure them.

    :rtype: :class:`~requests.Response`
    :returns: The "200 OK" response object returned after the multipart
              upload request.
    :raises: :exc:`ValueError` if ``size`` is not :data:`None` but the
             ``stream`` has fewer than ``size`` bytes remaining.
    """
    if size is None:
        data = stream.read()
    else:
        data = stream.read(size)
        if len(data) < size:
            msg = _READ_LESS_THAN_SIZE.format(size, len(data))
            raise ValueError(msg)

    client = self._require_client(client)
    transport = self._get_transport(client)
    if "metadata" in self._properties and "metadata" not in self._changes:
        self._changes.add("metadata")
    info = self._get_upload_arguments(client, content_type)
    headers, object_metadata, content_type = info

    hostname = _get_host_name(client._connection)
    base_url = _MULTIPART_URL_TEMPLATE.format(
        hostname=hostname, bucket_path=self.bucket.path, api_version=_API_VERSION
    )
    name_value_pairs = []

    if self.user_project is not None:
        name_value_pairs.append(("userProject", self.user_project))

    # When a Customer Managed Encryption Key is used to encrypt Cloud Storage object
    # at rest, object resource metadata will store the version of the Key Management
    # Service cryptographic material. If a Blob instance with KMS Key metadata set is
    # used to upload a new version of the object then the existing kmsKeyName version
    # value can't be used in the upload request and the client instead ignores it.
    if (
        self.kms_key_name is not None
        and "cryptoKeyVersions" not in self.kms_key_name
    ):
        name_value_pairs.append(("kmsKeyName", self.kms_key_name))

    if predefined_acl is not None:
        name_value_pairs.append(("predefinedAcl", predefined_acl))

    if if_generation_match is not None:
        name_value_pairs.append(("ifGenerationMatch", if_generation_match))

    if if_generation_not_match is not None:
        name_value_pairs.append(("ifGenerationNotMatch", if_generation_not_match))

    if if_metageneration_match is not None:
        name_value_pairs.append(("ifMetagenerationMatch", if_metageneration_match))

    if if_metageneration_not_match is not None:
        name_value_pairs.append(
            ("ifMetaGenerationNotMatch", if_metageneration_not_match)
        )

    upload_url = _add_query_parameters(base_url, name_value_pairs)
    upload = MultipartUpload(upload_url, headers=headers, checksum=checksum)

    upload._retry_strategy = _api_core_retry_to_resumable_media_retry(
        retry, num_retries
    )


  response = upload.transmit(


        transport, data, object_metadata, content_type, timeout=timeout
    )

google/cloud/storage/blob.py:1889:

self = <google.resumable_media.requests.upload.MultipartUpload object at 0x7f73d65324c0>

transport = <google.auth.transport.requests.AuthorizedSession object at 0x7f73d813c4c0>

data = b'y\x99\xf5\t\xc5\xc2\xc8N6\xf7D.\xce\xeey-\x14IU\xcdPf\xdd\x85\x8cV\xd3.\xeb(\xd8ULF|O\x97\xa7-\xa8!\xb4&\x9e\xe8Q\xe...82\xf2\x9f\xf7\x965\xd7w;.\x94L\x8a\xeb!w8U\xef\x9a+\xbe\x1d\x0b\x98\x94\x8d\xdc\xcd{k\x08\x93\xd4\xe9\x06\xd1\xb9\x7f'

metadata = {'name': 'BigFile07c10a4e8951433bb7bb8658d5c83c62'}

content_type = 'application/zip', timeout = 60
def transmit(
    self,
    transport,
    data,
    metadata,
    content_type,
    timeout=(
        _request_helpers._DEFAULT_CONNECT_TIMEOUT,
        _request_helpers._DEFAULT_READ_TIMEOUT,
    ),
):
    """Transmit the resource to be uploaded.

    Args:
        transport (~requests.Session): A ``requests`` object which can
            make authenticated requests.
        data (bytes): The resource content to be uploaded.
        metadata (Mapping[str, str]): The resource metadata, such as an
            ACL list.
        content_type (str): The content type of the resource, e.g. a JPEG
            image has content type ``image/jpeg``.
        timeout (Optional[Union[float, Tuple[float, float]]]):
            The number of seconds to wait for the server response.
            Depending on the retry strategy, a request may be repeated
            several times using the same timeout each time.

            Can also be passed as a tuple (connect_timeout, read_timeout).
            See :meth:`requests.Session.request` documentation for details.

    Returns:
        ~requests.Response: The HTTP response returned by ``transport``.
    """
    method, url, payload, headers = self._prepare_request(
        data, metadata, content_type
    )

    # Wrap the request business logic in a function to be retried.
    def retriable_request():
        result = transport.request(
            method, url, data=payload, headers=headers, timeout=timeout
        )

        self._process_response(result)

        return result


  return _request_helpers.wait_and_retry(


        retriable_request, self._get_status_code, self._retry_strategy
    )

.nox/system-3-8/lib/python3.8/site-packages/google/resumable_media/requests/upload.py:153:

func = <function MultipartUpload.transmit..retriable_request at 0x7f73d6597a60>

get_status_code = <function RequestsMixin._get_status_code at 0x7f73d865b5e0>

retry_strategy = <google.resumable_media.common.RetryStrategy object at 0x7f73d65324f0>
def wait_and_retry(func, get_status_code, retry_strategy):
    """Attempts to retry a call to ``func`` until success.

    Expects ``func`` to return an HTTP response and uses ``get_status_code``
    to check if the response is retry-able.

    ``func`` is expected to raise a failure status code as a
    common.InvalidResponse, at which point this method will check the code
    against the common.RETRIABLE list of retriable status codes.

    Will retry until :meth:`~.RetryStrategy.retry_allowed` (on the current
    ``retry_strategy``) returns :data:`False`. Uses
    :func:`_helpers.calculate_retry_wait` to double the wait time (with jitter)
    after each attempt.

    Args:
        func (Callable): A callable that takes no arguments and produces
            an HTTP response which will be checked as retry-able.
        get_status_code (Callable[Any, int]): Helper to get a status code
            from a response.
        retry_strategy (~google.resumable_media.common.RetryStrategy): The
            strategy to use if the request fails and must be retried.

    Returns:
        object: The return value of ``func``.
    """
    total_sleep = 0.0
    num_retries = 0
    # base_wait will be multiplied by the multiplier on the first retry.
    base_wait = float(retry_strategy.initial_delay) / retry_strategy.multiplier

    # Set the retriable_exception_type if possible. We expect requests to be
    # present here and the transport to be using requests.exceptions errors,
    # but due to loose coupling with the transport layer we can't guarantee it.

    while True:  # return on success or when retries exhausted.
        error = None
        try:
            response = func()
        except _CONNECTION_ERROR_CLASSES as e:
            error = e  # Fall through to retry, if there are retries left.
        except common.InvalidResponse as e:
            # An InvalidResponse is only retriable if its status code matches.
            # The `process_response()` method on a Download or Upload method
            # will convert the status code into an exception.
            if get_status_code(e.response) in common.RETRYABLE:
                error = e  # Fall through to retry, if there are retries left.
            else:
                raise  # If the status code is not retriable, raise w/o retry.
        else:
            return response

        base_wait, wait_time = _helpers.calculate_retry_wait(
            base_wait, retry_strategy.max_sleep, retry_strategy.multiplier
        )
        num_retries += 1
        total_sleep += wait_time

        # Check if (another) retry is allowed. If retries are exhausted and
        # no acceptable response was received, raise the retriable error.
        if not retry_strategy.retry_allowed(total_sleep, num_retries):


          raise error


.nox/system-3-8/lib/python3.8/site-packages/google/resumable_media/requests/_request_helpers.py:171:

func = <function MultipartUpload.transmit..retriable_request at 0x7f73d6597a60>

get_status_code = <function RequestsMixin._get_status_code at 0x7f73d865b5e0>

retry_strategy = <google.resumable_media.common.RetryStrategy object at 0x7f73d65324f0>
def wait_and_retry(func, get_status_code, retry_strategy):
    """Attempts to retry a call to ``func`` until success.

    Expects ``func`` to return an HTTP response and uses ``get_status_code``
    to check if the response is retry-able.

    ``func`` is expected to raise a failure status code as a
    common.InvalidResponse, at which point this method will check the code
    against the common.RETRIABLE list of retriable status codes.

    Will retry until :meth:`~.RetryStrategy.retry_allowed` (on the current
    ``retry_strategy``) returns :data:`False`. Uses
    :func:`_helpers.calculate_retry_wait` to double the wait time (with jitter)
    after each attempt.

    Args:
        func (Callable): A callable that takes no arguments and produces
            an HTTP response which will be checked as retry-able.
        get_status_code (Callable[Any, int]): Helper to get a status code
            from a response.
        retry_strategy (~google.resumable_media.common.RetryStrategy): The
            strategy to use if the request fails and must be retried.

    Returns:
        object: The return value of ``func``.
    """
    total_sleep = 0.0
    num_retries = 0
    # base_wait will be multiplied by the multiplier on the first retry.
    base_wait = float(retry_strategy.initial_delay) / retry_strategy.multiplier

    # Set the retriable_exception_type if possible. We expect requests to be
    # present here and the transport to be using requests.exceptions errors,
    # but due to loose coupling with the transport layer we can't guarantee it.

    while True:  # return on success or when retries exhausted.
        error = None
        try:


          response = func()


.nox/system-3-8/lib/python3.8/site-packages/google/resumable_media/requests/_request_helpers.py:148:

def retriable_request():
    result = transport.request(
        method, url, data=payload, headers=headers, timeout=timeout
    )


  self._process_response(result)


.nox/system-3-8/lib/python3.8/site-packages/google/resumable_media/requests/upload.py:149:

self = <google.resumable_media.requests.upload.MultipartUpload object at 0x7f73d65324c0>

response = <Response [503]>
def _process_response(self, response):
    """Process the response from an HTTP request.

    This is everything that must be done after a request that doesn't
    require network I/O (or other I/O). This is based on the `sans-I/O`_
    philosophy.

    Args:
        response (object): The HTTP response object.

    Raises:
        ~google.resumable_media.common.InvalidResponse: If the status
            code is not 200.

    .. _sans-I/O: https://sans-io.readthedocs.io/
    """
    # Tombstone the current upload so it cannot be used again (in either
    # failure or success).
    self._finished = True


  _helpers.require_status_code(response, (http.client.OK,), self._get_status_code)


.nox/system-3-8/lib/python3.8/site-packages/google/resumable_media/_upload.py:114:

response = <Response [503]>, status_codes = (<HTTPStatus.OK: 200>,)

get_status_code = <function RequestsMixin._get_status_code at 0x7f73d865b5e0>

callback = <function do_nothing at 0x7f73d86590d0>
def require_status_code(response, status_codes, get_status_code, callback=do_nothing):
    """Require a response has a status code among a list.

    Args:
        response (object): The HTTP response object.
        status_codes (tuple): The acceptable status codes.
        get_status_code (Callable[Any, int]): Helper to get a status code
            from a response.
        callback (Optional[Callable]): A callback that takes no arguments,
            to be executed when an exception is being raised.

    Returns:
        int: The status code.

    Raises:
        ~google.resumable_media.common.InvalidResponse: If the status code
            is not one of the values in ``status_codes``.
    """
    status_code = get_status_code(response)
    if status_code not in status_codes:
        if status_code not in common.RETRYABLE:
            callback()


      raise common.InvalidResponse(


            response,
            "Request failed with status code",
            status_code,
            "Expected one of",
            *status_codes
        )

E           google.resumable_media.common.InvalidResponse: ('Request failed with status code', 503, 'Expected one of', <HTTPStatus.OK: 200>)
.nox/system-3-8/lib/python3.8/site-packages/google/resumable_media/_helpers.py:108: InvalidResponse
During handling of the above exception, another exception occurred:
shared_bucket = <Bucket: gcp-systest-1673330037529>, blobs_to_delete = []

file_data = {'big': {'hash': b'cEome4a+NYd7YIXzXQnR5Q==', 'path': '/tmpfs/src/github/python-storage/tests/data/five-point-one-mb-f...g'}, 'simple': {'hash': b'3Hkwjv2WvCnKjNR6Z3CboA==', 'path': '/tmpfs/src/github/python-storage/tests/data/simple.txt'}}
def test_blob_update_storage_class_large_file(
    shared_bucket, blobs_to_delete, file_data
):
    from google.cloud.storage import constants

    blob = shared_bucket.blob(f"BigFile{uuid.uuid4().hex}")

    info = file_data["big"]


  blob.upload_from_filename(info["path"])


tests/system/test_blob.py:1113:

google/cloud/storage/blob.py:2683: in upload_from_filename

self.upload_from_file(

google/cloud/storage/blob.py:2556: in upload_from_file

_raise_from_invalid_response(exc)

error = InvalidResponse('Request failed with status code', 503, 'Expected one of', <HTTPStatus.OK: 200>)
def _raise_from_invalid_response(error):
    """Re-wrap and raise an ``InvalidResponse`` exception.

    :type error: :exc:`google.resumable_media.InvalidResponse`
    :param error: A caught exception from the ``google-resumable-media``
                  library.

    :raises: :class:`~google.cloud.exceptions.GoogleCloudError` corresponding
             to the failed status code
    """
    response = error.response

    # The 'response.text' gives the actual reason of error, where 'error' gives
    # the message of expected status code.
    if response.text:
        error_message = response.text + ": " + str(error)
    else:
        error_message = str(error)

    message = f"{response.request.method} {response.request.url}: {error_message}"


  raise exceptions.from_http_status(response.status_code, message, response=response)


E       google.api_core.exceptions.ServiceUnavailable: 503 POST https://storage-preprod-test-unified.googleusercontent.com/upload/storage/v1_preprod/b/gcp-systest-1673330037529/o?uploadType=multipart: Service Unavailable: ('Request failed with status code', 503, 'Expected one of', <HTTPStatus.OK: 200>)
google/cloud/storage/blob.py:4368: ServiceUnavailable

[cojenco]

Downgraded to p2 and potentially closing. This looks like an issue caused by preprod being down https://testgrid.corp.google.com/bigstore#preprod