Max outbound streams is 100, 100 open #1414
Comments
|
@valiant1x Thanks for opening this, could you perhaps share the following... ?
|
florimondmanca
added
http/2
|
@florimondmanca Thanks for the quick reply. Traceback: I'll try to provide a simple repro, it's a little challenging because it seems to depend on our target website being in 'queue' mode. I don't want to share the target website here, but the gist of it is, we have a class that inherits class LuckyScraper(httpx.AsyncClient):
def __init__(self, **kwargs):
super().__init__(**kwargs)
async def get(self, url: str, **kwargs) -> object:
return await self.request("GET", url, **kwargs)
async def request(self, method: str, url: str, **kwargs) -> object:
attempts = 0
retries = 10
while attempts < retries:
res = await super().request(method, url, **kwargs)
if res and not res.is_error:
return res
attempts += 1
await asyncio.sleep(5)
sslCtx = getCustomSslCtx()
proxies = { ... }
async with LuckyScraper(verify=sslCtx, proxies=proxies, timeout=20, http2=True) as client:
headers = OrderedDict([
('accept', '*/*'),
#other headers
])
response = await client.get(url, headers=headers)During queue mode, we will see the worker spend a lot of time in Spawning 100 requests does not trigger this, only the retries in the same async context loop trigger the issue. Batches of tasks are sent to Are there streams opened during this process that are not gracefully cleaned up because the underlying httpx instance is long living? We need a persistent instance because we rely on some session cookies that must persist through the refreshes, in order for the queue system on target website to interact properly. |
|
@valiant1x When you write « queue mode », do you mean the server returning 503 Service Unavailable responses, which causes your custom get method to loop until it gets a proper non-error response? Or does it mean something else? |
The intended behavior would be that when FWIW our networking code for HTTP/2 lives here: https://github.com/encode/httpcore/blob/master/httpcore/_async/http2.py You’ll notice that there’s error handling around the h2 library telling us « I have no streams left », in which case we end up creating a new connection. The exception class there is Right now what I assume may be happening is some kind of bug (either is, h2, or the server) causes the stream to « leak » after the response is sent. h2 still considers it to be active, and so we keep opening more and we reach the 100 limit. |
Exactly, queue mode is 503. We simply retry here, for many attempts (200+)
Can we disable keep-alive on our retries to mitigate the issue, or forcefully close the underlying stream if the response status code is_error? I would think this is desired behavior, to close a stream on error. Testing was a mess today and the 503 is no longer triggering, but I thought I saw the same issue when specifying |
Well, there's not really an error here — the server responds just fine, it's just that we get a Some next-step things to figure out:
|
@valiant1x Disabling keep-alive would mean not reusing connections, so at this point you might as well create a new client instance each time, i.e. have (Btw, I wouldn't recommend subclassing from |
Great ideas. I'll get some testing going to reproduce 503s with local server and see if I can get anywhere.
If I did not need to preserve cookies, I would create a new instance every time as a hack workaround to this issue, but I have found cookie importing and exporting to be flaky so I want to avoid this. If you instantiate httpx with a cookie that the server then updates, it seems difficult to avoid duplicate cookies. Duplicate cookies are then sent in subsequent requests and also crash
Can you elaborate on why it would be better to use a class attribute instead of subclassing |
|
Reproducing this and finding the root cause has proved to be very time intensive. The error is sporadic and does not seem to depend much on server response code necessarily, but just on re-using an Here is a stack trace of a Connection lost exception. We are going to simply revert to HTTP/1.1 for now unfortunately. |
That's quite an interesting observation. Given how infrequent those are, and assuming they correspond to But… When that happens in the middle of a request, all we do is release the stream semaphore, then bubble up. But def send_headers(self, stream_id, headers, end_stream=False,
priority_weight=None, priority_depends_on=None,
priority_exclusive=None):
...
# Check we can open the stream.
if stream_id not in self.streams:
max_open_streams = self.remote_settings.max_concurrent_streams
if (self.open_outbound_streams + 1) > max_open_streams:
raise TooManyStreamsError(
"Max outbound streams is %d, %d open" %
(max_open_streams, self.open_outbound_streams)
)
@property
def closed(self):
"""
Whether the stream is closed.
"""
return self.state_machine.state == StreamState.CLOSEDSo, since we just give up and propagate the exception without letting So each time that So… Maybe we just have to call |
Great analysis and synthesis. It seems like a probable solution. I will re-test it at some point in the medium term future. If anyone else encounters the same problem in the meantime, they will have a good starting point for resolution. |
|
@valiant1x I pushed encode/httpcore#253 with a naive attempt at resolving this based on my previous answer. You can test it by installing HTTPCore from the branch. Would you be able to test it and see if it resolves the issue? pip install "git+https://github.com/encode/httpcore.git@fm/http2-close-conn#egg=httpcore" |
|
I have also encounter this issue. And it looks like it was a server-side limitation, in h2, My questions are:
|
|
I think the internal value of Maybe try to add a conditional variable (looks like we need to add a |
|
Hello there. I get this same error when too many connections get unexpectedly closed like the OP. @florimondmanca your patch doesn't seems to work with newer versions since I get an exception when the new line of code is executed. |
Checklist
master.Describe the bug
I tried looking through GitHub issues and documentation for an explanation of how to increase this limit but did not find any answers. We use
httpx.AsyncClient(http2=True)Async client with HTTP/2 in high concurrency scenarios, although I observed the same error withhttp2=False. It is necessary to keep many streams open when the target website goes into queue mode. At this point, if running more than 100 concurrent requests, we encounter this error. How can we resolve it?I have tried initializing the AsyncClient with
httpx.Limits(max_connections=None, max_keepalive_connections=None)but this did not make a difference.To reproduce
Will provide upon request, this seems more of a design/conceptual problem where reproducible POC may not be necessary
Expected behavior
Client can configure the maximum number of outbound streams.
Actual behavior
Client is provided with error about maximum streams and request fails.
Debugging material
n/a
Environment
Additional context
Issue was somewhat acknowledged in #832 but there was no resolution provided.
How can we increase the streams limit or otherwise work around the issue here?
The text was updated successfully, but these errors were encountered: