Quarkus Rest Client experiences massive slowdowns under concurrent load starting from Quarkus 3.13.0 #42855
Comments
|
/cc @cescoffier (rest-client), @geoand (kotlin,rest-client) |
|
Could you try if #42823 makes any difference for you? |
|
From what I can see, things are better with |
|
I'm currently building it on my machine. |
|
I think what's remaining might be due to #42858 but from what I can see the behavior is a lot saner in |
|
Hmm how would I use this locally? I did a |
|
You need to use: <dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-bom</artifactId>
<version>999-SNAPSHOT</version>
<type>pom</type>
<scope>import</scope>
</dependency>as the BOM |
|
For me 999-SNAPSHOT (avg 580ms) is slightly faster than 3.11.3 (avg 620ms). So it would seem the patches have helped with the problem and even slightly improved the speed (at least on my machine). |
|
Cool, thanks a lot for checking! Given that #42823 is already marked for backport, let's go ahead and close this. |
Describe the bug
We have a project were we use the Quarkus Rest Client to call other services within a service of our own. We use a simple RestClient like this:
We then call this Rest client from within a controller of our service:
When we have a sufficient amount of concurrent calls with this rest client, we experience massive slowdowns starting with Quarkus 3.13.0, e.g. calls that take 300-400ms usually start to take 4000-5000ms under load. The behaviour is only reproducable when putting some load on the client and only in Quarkus versions 3.13.0 or later. In our specific case we migrated from version 3.11.3 to 3.13.0, but i have tested all versions up to 3.13.0. The last "good" version was 3.12.3.
Expected behavior
The client's response times do not deterioate under heavy load, given that the external system responds in a timely fashion.
Actual behavior
The client's response times become much slower the more concurrent calls the client is executing.
Output from 3.13.3 under heavy load (50 concurrent calls, note how times deteriorate as more requests are in-flight)
Output from 3.11.3 (50 concurrent calls, also slight deterioration but may be caused by the upstream server not responding fast enough)
How to Reproduce?
Reproducer:
https://github.com/derkork/quarkus-performance-issue-reproducer
The project consists of a minimal server, client and a test. The test fires up a WireMock Server which simulates 200ms delay of the upstream system and then runs 10 rounds of 50 concurrent requests against the Quarkus application. The test then asserts that the average response time is below 1000ms.
The version that is commited is set up to use Quarkus 3.13.3 and in this version the test should fail. If you change the Quarkus version in the
pom.xmlto 3.11.3 the test works and the response times significantly improve.Output of
uname -aorverDarwin .local 23.5.0 Darwin Kernel Version 23.5.0: Wed May 1 20:09:52 PDT 2024; root:xnu-10063.121.3~5/RELEASE_X86_64 x86_64
Output of
java -versionopenjdk version "22.0.2" 2024-07-16
Quarkus version or git rev
3.13.0 or later
Build tool (ie. output of
mvnw --versionorgradlew --version)Apache Maven 3.9.8 (36645f6c9b5079805ea5009217e36f2cffd34256)
Additional information
This also happens in the native build of the software running in a docker container, it is not limited to OSX. We have seen this in production under heavy load. It doesn't seem to be consistently reproducible in every environment though.
The text was updated successfully, but these errors were encountered: