ubi-quarkus-mandrel-builder-image is slower than ubi-quarkus-graalvmce-builder-image

[rsvoboda]

Describe the bug

I noticed that ubi-quarkus-mandrel-builder-image is slower than ubi-quarkus-graalvmce-builder-image

The difference is quite noticeable, in https://github.com/quarkus-qe/quarkus-test-suite daily runs I see difference of 1 hour for DB related modules when running in native.

When using quay.io/quarkus/ubi-quarkus-graalvmce-builder-image:22.3-java17 the execution time is ~4:40h
When using quay.io/quarkus/ubi-quarkus-mandrel-builder-image:22.3-java17 the execution time is ~5:45h
run link: https://github.com/quarkus-qe/quarkus-test-suite/actions/runs/4672855488

So there is 1 hour difference and that's a noticeable diff.

I do see a diff even with simple jaxrs sample app from code.quarkus:

Finished generating 'code-with-quarkus-1.0.0-SNAPSHOT-runner' in 1m 49s. (graalvmce)
  vs.
Finished generating 'code-with-quarkus-1.0.0-SNAPSHOT-runner' in 2m 3s.  (mandrel)

Expected behavior

Mandrel builder image is as fast as GraalVM based builder image

Actual behavior

GraalVM based builder image is faster than Mandrel builder image

How to Reproduce?

• get the app from https://code.quarkus.io/?S=io.quarkus.platform%3A3.0
• run the build with graalvmce-builder-image:
  • mvn clean package -DskipTests -Dnative -Dquarkus.native.container-build=true -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-graalvmce-builder-image:22.3-java17
• run the build with mandrel-builder-image
  • mvn clean package -DskipTests -Dnative -Dquarkus.native.container-build=true -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel-builder-image:22.3-java17
• compare the times needed to build the native image
  • search for Finished generating 'code-with-quarkus-1.0.0-SNAPSHOT-runner' in

Output of uname -a or ver

macOS

Output of java -version

Java 17

GraalVM version (if different from Java)

Mandrel and GraalVMCE 22.3 images

Quarkus version or git rev

Quarkus 3.0.0.CR2

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot]

/cc @Karm (mandrel), @galderz (mandrel), @zakkak (mandrel)

[rsvoboda]

@Karm / @galderz any initial observations about this issue?

[Karm]

It is in the pipeline, we have some theories to test.

[rsvoboda]

@Karm / @zakkak any news on this topic?

[zakkak]

Hi @rsvoboda , we have investigated this in the past and concluded that the main cause of the difference is that:

• GraalVM CE is using an ahead of time compiled version of Graal JIT (libgraal) when both for JIT compiling the analysis code and generating the native code
• Mandrel, on the other hand, uses the default OpenJDK JIT compilers (C1/C2) to JIT compile the analysis code and the Graal compiler (not ahead of time compiled) to generate the native code.

According to my non-exhaustive measurements the overall performance difference is around 10%. Since github runners are not very stable for performance measurements it would be great if you could name a couple of heavily affected tests, to see if there is anything additional going on for this cases.

[rsvoboda]

Thanks @zakkak. I noticed diff ~12 %, not a perf hardware.

Is there anything that could close the gap between Mandrel and GraalVM CE?
Or it's the architecture choice (Mandrel is based on top of OpenJDK) that prevents to get on par with Graal JIT?

[zakkak]

Is there anything that could close the gap between Mandrel and GraalVM CE?
Or it's the architecture choice (Mandrel is based on top of OpenJDK) that prevents to get on par with Graal JIT?

Correct, it's mainly the architecture/design choice that prevents us from getting on par with GraalVM CE builds.
In theory if we better understand the source of the main overhead we could potentially do some tuning (it's another Java application after all). FTR @jerboaa suggested at some point disabling some compilation tiers, but toying a bit with various related options I didn't see any significant gains.

[gsmet]

I will close this one as it's not a Quarkus issue.