http2 connection is not renewed after server closes?

[twisteroidambassador]

Using AsyncClient with http2=True, after sending many requests, all subsequent requests fail immediately with an exception like this: httpx.RemoteProtocolError: <ConnectionTerminated error_code:ErrorCodes.NO_ERROR, last_stream_id:19999, additional_data:None>

It feels like the server has closed the connection after serving a fixed number of request (20000?), but httpx keeps trying to use the same connection, without attempting to create a new one.

Here's the full traceback:

Traceback (most recent call last):
  File "~\venv\lib\site-packages\h2\connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~\venv\lib\site-packages\httpcore\_async\http2.py", line 105, in handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "~\venv\lib\site-packages\httpcore\_async\http2.py", line 202, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "~\venv\lib\site-packages\h2\connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "~\venv\lib\site-packages\h2\connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~\venv\lib\site-packages\httpx\_transports\default.py", line 60, in map_httpcore_exceptions
    yield
  File "~\venv\lib\site-packages\httpx\_transports\default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "~\venv\lib\site-packages\httpcore\_async\connection_pool.py", line 253, in handle_async_request
    raise exc
  File "~\venv\lib\site-packages\httpcore\_async\connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "~\venv\lib\site-packages\httpcore\_async\connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "~\venv\lib\site-packages\httpcore\_async\http2.py", line 138, in handle_async_request
    raise RemoteProtocolError(self._connection_error_event)
httpcore.RemoteProtocolError: <ConnectionTerminated error_code:ErrorCodes.NO_ERROR, last_stream_id:19999, additional_data:None>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~\decryptor.py", line 138, in do_decrypt_work
    resp = await client.get(test_v3_url)
  File "~\venv\lib\site-packages\httpx\_client.py", line 1729, in get
    return await self.request(
  File "~\venv\lib\site-packages\httpx\_client.py", line 1506, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "~\venv\lib\site-packages\httpx\_client.py", line 1593, in send
    response = await self._send_handling_auth(
  File "~\venv\lib\site-packages\httpx\_client.py", line 1621, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "~\venv\lib\site-packages\httpx\_client.py", line 1658, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "~\venv\lib\site-packages\httpx\_client.py", line 1695, in _send_single_request
    response = await transport.handle_async_request(request)
  File "~\venv\lib\site-packages\httpx\_transports\default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "AppData\Local\Programs\Python\Python310\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "~\venv\lib\site-packages\httpx\_transports\default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: <ConnectionTerminated error_code:ErrorCodes.NO_ERROR, last_stream_id:19999, additional_data:None>

Package versions:
h2-4.1.0
httpcore-0.14.7
httpx-0.22.0

This may be related? encode/httpcore#45

[adriangb]

I've run into this as well. I'm just disabling http2 for now, which is not great. I'll try to look into causes

[set92]

2023 and still here boys! :D

PrefectHQ/prefect#7442

I haven't seen a workaround, so I suppose I'll disable http2 for now, which sucks but seems to be the only way to fix it.

[akdienes]

I am also hitting this.

[python273]

Same error

code: https://github.com/python273/eimi/blob/5fc4a1744191a5954955d77786e94bcaf8d6d5de/main.py
tracebacks: https://gist.githubusercontent.com/python273/651744e3e7140a3ed012661922ab075c/raw/c0bac785225b6ec475f6e8c6f0c913f60c534a1b/httpx_error.log

[zanieb]

I have a reproduction at https://github.com/madkinsz/httpcore-h2-closed and a draft fix at encode/httpcore#679

[clairem-sl]

I've been having the same issue. How to work around this?

OS: Windows 10 Pro 64-bit
Python: CPython 3.11
h2==4.1.0
httpcore==0.17.2
httpx==0.24.1

Edit: I've been looking around httpx documentation, there doesn't seem a way to trigger httpx.AsynClient to just invalidate its connection pool and rebuild a new pool. If there's a way to do that, I can have the first asyncio Task that sees ConnectionTerminated to trigger that invalidation+rebuilding process...

[python273]

I think there was a fix in httpcore 0.17.1

[joemirizio]

I upgraded to the latest versions of httpx and httpcore and still get a ConnectionTerminated. My workaround has also been disabling http2.

Python: CPython 3.11.2
h2==4.1.0
httpcore==0.17.2
httpx==0.24.1

Traceback

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
                         ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http2.py", line 119, in handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http2.py", line 216, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "/usr/local/lib/python3.11/site-packages/h2/connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "/usr/local/lib/python3.11/site-packages/h2/connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 261, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection.py", line 96, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpcore/_async/http2.py", line 152, in handle_async_request
    raise RemoteProtocolError(self._connection_error_event)
httpcore.RemoteProtocolError: <ConnectionTerminated error_code:0, last_stream_id:19999, additional_data:None>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspaces/mycode/loader.py", line 69, in process_table_package
    package = await vendor.get_table_package(table_id)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/mycode/vendor.py", line 309, in get_table_package
    table, columns = await asyncio.gather(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/tasks.py", line 684, in _wrap_awaitable
    return (yield from awaitable.__await__())
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiostream/core.py", line 33, in wait_stream
    async for item in streamer:
  File "/usr/local/lib/python3.11/site-packages/aiostream/stream/aggregate.py", line 71, in list
    async for item in streamer:
  File "/workspaces/mycode/vendor.py", line 260, in get_column_objects
    column_json["_deprecated"] = await self.is_object_deprecated(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/mycode/vendor.py", line 290, in is_object_deprecated
    flags = await self.get_flag_objects(oid, otype)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/mycode/vendor.py", line 282, in get_flag_objects
    flags = await stream.list(self.get_flags(oid, otype))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiostream/core.py", line 33, in wait_stream
    async for item in streamer:
  File "/usr/local/lib/python3.11/site-packages/aiostream/stream/aggregate.py", line 71, in list
    async for item in streamer:
  File "/workspaces/mycode/vendor.py", line 72, in _api_get_paginated
    resp = await self._api_get(
           ^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/mycode/client.py", line 57, in _api_get
    return await self._api_request("GET", endpoint, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/mycode/client.py", line 52, in _api_request
    response = await self.client.request(method, f"{self.uri}/{endpoint}", **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: <ConnectionTerminated error_code:0, last_stream_id:19999, additional_data:None>

[zanieb]

The previous fix just made it so you can catch this error and retry the request yourself without it failing again.

encode/httpcore#683 is needed to automatically retry the request.

[joemirizio]

Gotcha - thanks for the clarification!

[khalezin]

We had the same problem. We have following configuration
proxy(nginx) > uvicorn
We fixed the issue by increasing this value on the side of the:

keepalive_requests 1000;

http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests