Worse than expected performance with native compared to JVM

[stianst]

Describe the bug

I'm doing a PoC for a native runtime of Keycloak, and have observed much worse performance with native mode compared to JVM mode than I would have expected.

I've stripped Keycloak down to the bear bones for a OIDC use-case for this purpose as I was also looking for what a very small distribution of Keycloak would look like, not just the benefits of native vs jvm. The PoC is available here.

For those familiar with Keycloak the key differences here are:

• Removed features such as SAML, LDAP, embedded Infinispan caches
• Only support external PostgreSQL database

Startup time and memory usage of native mode is impressive, while throughout and latency is a lot worse than I was hoping for. I appreciate there is a performance impact of native vs JVM, but this is a lot higher than I was expecting.

From a 10 min run on my laptop with a local PostgreSQL database I get the following results with 10 concurrent users sending requests:

Native (min=15, 95th=29, max=64):

JVM (min=8, 95th=20, max=282):

Couple things to observe here:

• Latency is almost 2x on native vs JVM
• Latency is very stable with JVM, with only some occasional spikes (probably GC), while on native there's constantly a big difference between the request times

I opened this as a bug, but perhaps it could be converted to a discussion, as I'm not sure it's something on my end, or if this is an issue in Quarkus. I've also done no tuning of native, pretty much just using whatever Quarkus comes with by default.

Expected behavior

Latency/throughput is stable in native mode, and performance is closer to JVM mode.

Actual behavior

Big difference between latency in native and JVM modes, and native mode giving very different response times for parallel requests.

How to Reproduce?

No response

Output of uname -a or ver

Linux fedora 6.4.10-200.fc38.x86_64

Output of java -version

openjdk version "17.0.8" 2023-07-18

GraalVM version (if different from Java)

OpenJDK 64-Bit Server VM Mandrel-23.0.0.0-Final

Quarkus version or git rev

3.2.4.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.2

Additional information

No response

[geoand]

cc @galderz @zakkak

[galderz]

@stianst You should go through the steps in https://quarkus.io/guides/native-reference#profiling and see if anything stands out in the profiling data you obtain (e.g. flamegraphs). E.g. when we did this for Infinispan server, we noticed that in native trace logging was enabled by default which lead to really bad performance when comparing both. Share the data when you have it.

[franz1981]

I cannot see the rate and CPU usage of the 2s. Additionally to the comment made by Galder I suggest to not use the default perf event but -e cpu-clock with a decent frequency -F 100 (not more or the box will die)

[maxandersen]

moved to discussion as no exact issue identified (yet).

@galderz recommends looking at https://quarkus.io/guides/native-reference#profiling - that should give you some ideas on where things are stealing time.

[franz1981]

I think, with @galderz we have found few reasons why this is happening ;)