Improve robustness for HTTP/2

[tomchristie]

We've got a handful of open issues around HTTP/2 robustness.

I'm going to collate these into a single issue, to help us keep the issue easily reviewable & improve prioritisation.

• "Server disconnected" on HTTP/2 connection pool with big keepalive_expiry. #2983
• httpx + http2 raises a lot of exceptions when parallelized with ThreadPoolExecuton #3002
• HTTP 2.0 Throws KeyError rather than the internal exception thrown in the thread #3072

[justin-jeffery-ipf]

I just had a customer experience the following issue. No async is being used by our product at this time.

When I try my code with a try/except and log the process, I see the same error occurring the same; however, the stream id is reset back to 1, and the data is properly processed.

Traceback (most recent call last):
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_transports\default.py", line 72, in map_httpcore_exceptions
    yield
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_transports\default.py", line 236, in handle_request
    resp = self._pool.handle_request(req)
  File "C:\Data\python_test\venv\lib\site-packages\httpcore\_sync\connection_pool.py", line 216, in handle_request
    raise exc from None
  File "C:\Data\python_test\venv\lib\site-packages\httpcore\_sync\connection_pool.py", line 196, in handle_request
    response = connection.handle_request(
  File "C:\Data\python_test\venv\lib\site-packages\httpcore\_sync\connection.py", line 101, in handle_request
    return self._connection.handle_request(request)
  File "C:\Data\python_test\venv\lib\site-packages\httpcore\_sync\http2.py", line 185, in handle_request
    raise exc
  File "C:\Data\python_test\venv\lib\site-packages\httpcore\_sync\http2.py", line 148, in handle_request
    status, headers = self._receive_response(
  File "C:\Data\python_test\venv\lib\site-packages\httpcore\_sync\http2.py", line 292, in _receive_response
    event = self._receive_stream_event(request, stream_id)
  File "C:\Data\python_test\venv\lib\site-packages\httpcore\_sync\http2.py", line 333, in _receive_stream_event
    self._receive_events(request, stream_id)
  File "C:\Data\python_test\venv\lib\site-packages\httpcore\_sync\http2.py", line 352, in _receive_events
    raise RemoteProtocolError(self._connection_terminated)
httpcore.RemoteProtocolError: <ConnectionTerminated error_code:ErrorCodes.NO_ERROR, last_stream_id:1999, additional_data:None>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "C:\Data\python_test\ipf_stream_test.py", line 25, in <module>
    get_interfaces_from_json(intent['sn'])
  File "C:\Data\python_test\ipf_stream_test.py", line 16, in get_interfaces_from_json
    json_log = json.loads(device.get_json())
  File "C:\Data\python_test\venv\lib\site-packages\ipfabric\models\device.py", line 124, in get_json
    return raise_for_status(self.client.get(f"/snapshots/{_}/devices/{encoded_sn}/json")).text
  File "C:\Data\python_test\venv\lib\site-packages\ipfabric\api.py", line 142, in get
    return self._client.get(url, *args, params=params, **kwargs)
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_client.py", line 1066, in get
    return self.request(
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_client.py", line 837, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_client.py", line 926, in send
    response = self._send_handling_auth(
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_client.py", line 954, in _send_handling_auth
    response = self._send_handling_redirects(
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_client.py", line 991, in _send_handling_redirects
    response = self._send_single_request(request)
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_client.py", line 1027, in _send_single_request
    response = transport.handle_request(request)
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_transports\default.py", line 235, in handle_request
    with map_httpcore_exceptions():
  File "C:\Users\jdro\AppData\Local\Programs\Python\Python310\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Data\python_test\venv\lib\site-packages\httpx\_transports\default.py", line 89, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: <ConnectionTerminated error_code:ErrorCodes.NO_ERROR, last_stream_id:1999, additional_data:None>

It seems that once the stream ID hits 1999 it fails and then resets back to 1.

[mvasilkov]

Can confirm, consistently getting httpx.RemoteProtocolError: <ConnectionTerminated error_code:0, last_stream_id:1999, additional_data:None> with http2.

[tomchristie]

Initial step in resolving that is reproducibility.

Does anyone have time to run @jonathanslenders example here #2983 and confirm that reproduces the issue?