HTTP/2 max local stream count exceeded #6603
Comments
|
Below is the Exception Stack trace |
|
@prakashnandihal what's your configuration for Do your request have a request body? Do your request have a total timeout (i.e. Do you have other failures in your client-side logs, like |
|
Actually, can you attach a reproducible Maven project that shows the issue if it's that simple for you to reproduce? |
|
@sbordet Please find the answers inline. Do your request have a request body? Do your request have a total timeout (i.e. request.timeout(...))? Do your request have a total timeout (i.e. request.timeout(...))? can you attach a reproducible Maven project that shows the issue if it's that simple for you to reproduce? Additional Logs : |
|
joakime
changed the title
sbordet
changed the title
|
@joakime I changed the title back to original. To what server the client is connecting to is irrelevant, this is a client issue. |
|
So if I'm reading between the lines of what @joakime and @sbordet are saying.... the issue might be that the APN server is setting the max streams per connection to 1 (which is the limit that we see in the exception) in such a way that it causes a race in our client connection pool. The client should never see this exception because when the max streams is exceeded a new connection is created.... but there is a limit of 100 connections per destination. So perhaps the issue is that once those 100 connections are used up, a request is getting past the point where it should queue waiting for a connection to become available and actually attempts to create a stream on an existing connection - this creation races with the server setting the limit to 1, so the limit is exceeded. So maybe we should have a similar test where the server always sets the stream limit to 1 whilst the client is receiving many requests for a connection limited destination? |
|
@sbordet : I have attached maven project for your reference. As per apple developer documentation , APNS supports multiple concurrent streams and number of concurrent streams varies depending upon the server load. Can you please explain what is wrong with client? |
|
@prakashnandihal the token is already expired, unfortunately, so I just get 403. Can you please run your test project with the system property |
This comment has been minimized.
This comment has been minimized.
|
@prakashnandihal trying. |
|
@sbordet |
Made MAX_CONCURRENT_STREAMS setting work on a per-connection basis. Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
|
@prakashnandihal I think we have found the problem. It should be fixed in this branch: |
|
@sbordet yes i will try After cloning jetty-9.4.x-6603-max-concurrent-streams-exceeded branch, i hope below steps are correct to test the fix, please confirm |
|
These are the commands ... $ git clone https://github.com/eclipse/jetty.project.git
$ cd jetty.project
$ git checkout jetty-9.4.x-6603-max-concurrent-streams-exceeded
$ mvn clean install -DskipTestsThen use 9.4.44-SNAPSHOT in your project. |
|
@joakime : Thanks for providing info. @sbordet : Issue is Fixed. I tested fix from jetty-9.4.x-6603-max-concurrent-streams-exceeded branch, i did not see this error "max local stream count exceeded". When will this fix publicly available ? When i set High value like 500 or 1000 to maxConnectionsPerDestination, i got below intermittent errors I have below couple of questions on Jetty Http2 API. Please provide your inputs Currently we are using jetty Http2 API for sending push notifications in production and Now And also please recommend us what are all configuration parameters we can tweak to enhance the throughput of http2client. And also I want to Understand more on MaxRequestsQueuedPerDestination and MaxConnectionsPerDestination properties of HttpClient object.
Example: 2)MaxConnectionsPerDestination: |
This is not recommended. |
@joakime |
|
You still don't need multiple You should be able to use the Request tagging for those 5 applications. (see |
Thanks for the feedback!
With Jetty 9.4.44, the date is not set yet.
All of them seem caused by an underconfigured machine (a machine that is too small or configured too small for the job).
We cannot tell you the max throughput, you have to measure it. With multiple machines will be easier, provided the network is not a bottleneck. At 80k requests/s you saturate a 1 Gib network with about
One queue per destination.
Correct.
This is currently simpler than using the With 5 |
Updates after review. Updated the maxMultiplex mechanism to always work on Pool.Entry, rather than on Pool. Updated Pool javadocs. Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
Fixed javadocs. Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
|
@sbordet Thanks For Your Info. |
|
@prakashnandihal we are still discussing what is the best fix for this issue, so we don't have a date yet. |
Made MAX_CONCURRENT_STREAMS setting work on a per-connection basis. Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
Updates after review. Updated the maxMultiplex mechanism to always work on Pool.Entry, rather than on Pool. Updated Pool javadocs. Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
Fixed javadocs. Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
Incorporated changes from #6648. Now the maxMultiple value is pulled from its primary value, i.e. HTTP2Session.maxLocalStreams, rather than being set in multiple places. Deprecated usages of maxMultiplex and maxUsageCount. Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
Added comment as suggested in review. Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
|
@sbordet : We have to release fix for this bug to production Pod, Please notify us the once the release date of 9.4.44 is finalized. so that We will plan release of this fix to production pod. |
Made newEntry() private to avoid creation of more than 2 Entry subclasses. Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
|
@sbordet what is the release date of 9.4.44? |
Jetty version(s) 9.4.41.v20210516
Java version/vendor 1.8.0_271
OS type/version: Linux and Windows 10 machine
Description
HTTP/2 max local stream count exceeded
We were using jetty 9.4.35 version and we encountered this "HTTP/2 max local stream count 1 exceeded" error ,Simon suggested us to upgrade jetty version to 9.4.41 , so we did jetty jar upgrade to 9.4.41 version , but still we see the error .
Below is the code snippet we have written to create http2 client to post requests to destination host.
How to reproduce?
Set the MaxConnectionsPerDestination value to 100 and try to post more than 10K requests.
The text was updated successfully, but these errors were encountered: