Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempting to re-establish stream for both 470 and 471 #391

Open
barendatnicolab opened this issue Apr 9, 2024 · 1 comment
Open

Attempting to re-establish stream for both 470 and 471 #391

barendatnicolab opened this issue Apr 9, 2024 · 1 comment

Comments

@barendatnicolab
Copy link

I am running this emulator using testcontainers. If I have a couple tests running in parallel, I run into the following obscure error

--- Logging error ---
Traceback (most recent call last):
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/logging/__init__.py", line 1086, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/threading.py", line 937, in _bootstrap
    self._bootstrap_inner()
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/grpc/_channel.py", line 1731, in channel_spin
    call_completed = event.tag(event)
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/grpc/_channel.py", line 244, in handle_event
    callback()
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/google/api_core/bidi.py", line 459, in _on_call_done
    self._reopen()
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/google/api_core/bidi.py", line 494, in _reopen
    _LOGGER.info("Re-established stream")
Message: 'Re-established stream'
Arguments: ()
Exception in thread Thread-LeaseMaintainer:
Traceback (most recent call last):
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/grpc/_channel.py", line 1176, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/grpc/_channel.py", line 1005, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:55801: Failed to connect to remote host: Connection refused"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-04-09T12:03:52.857744+02:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:55801: Failed to connect to remote host: Connection refused"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target
    result = target()
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:55801: Failed to connect to remote host: Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/leaser.py", line 201, in maintain_leases
    expired_ack_ids = self._manager._send_lease_modacks(
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py", line 1047, in _send_lease_modacks
    self._dispatcher.modify_ack_deadline(items, ack_deadline)
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/dispatcher.py", line 358, in modify_ack_deadline
    _, requests_to_retry = self._manager.send_unary_modack(
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py", line 726, in send_unary_modack
    self._client.modify_ack_deadline(
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/google/pubsub_v1/services/subscriber/client.py", line 1487, in modify_ack_deadline
    rpc(
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func
    return retry_target(
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target
    _retry_error_helper(
  File "/Users/johndoe/miniconda3/envs/my-service/lib/python3.9/site-packages/google/api_core/retry/retry_base.py", line 221, in _retry_error_helper
    raise final_exc from source_exc
google.api_core.exceptions.RetryError: Timeout of 60.0s exceeded, last exception: 503 failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:55801: Failed to connect to remote host: Connection refused

It keeps attempting to re-establish the stream, and I have to manually abort the tests run. I am experiencing this issue on 470 and 471. When reverting to 469, the issue disappears.

@catchyzheng
Copy link
Member

Hello, do you still have the issue on the latest gcloud v490?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants