Investigate suboptimal score on TechEmpower benchmarks

[dreis2211]

Hi 👋

I'm not a particular huge fan of the TechEmpower benchmarks, because they barely reflect real use-cases, but (novice) programmers who maybe look for some resources that might help them with their questions of what framework to choose for their next projects might look into these benchmarks and make their decision (partly) based on these numbers.

I've compared Quarkus and Spring(-Boot) here to compare two popular frameworks that have similar goals and deliver similar functionality. There is also a comparison over at Baeldung https://www.baeldung.com/spring-boot-vs-quarkus that compares these two.

Imho, this particular comparison isn't exactly great news for the Spring ecosystem. While there are some benchmarks on par with Quarkus, the majority is "won" by Quarkus - and by a greater margin one would expect. I've recently overheard a conversation why one should use Spring if Quarkus delivers everything as well, but with better performance. Now, we all know that there is more to this story and application A might be better suited for Quarkus while application B might be better suited for Spring - for whatever reasons. There is no silver bullet, after all. But this conversation kept me thinking. Obviously, I'm biased towards Spring and I love the ecosystem, but I'm wondering if we can do something about the performance gap (in these benchmarks).

I know for example that recent changes to the Quarkus benchmarks have been done by people involved with Quarkus. The Spring benchmarks are somewhat driven infrequently by community members (I think I have done something on these ages ago myself as well).

I'm also happy to help with this and at least update the Spring(-Boot) one to latest versions but I wonder if you have comparable machines at hand that you could run these benchmarks against and find out why these numbers are so comparably "bad". (I do have the suspicion that the numbers are actually much closer in reality). I'm also happy to help with the investigations, if you want me to 😉

Cheers,
Christoph

P.S.: The point of this should be not to overtake Quarkus or establish a Spring vs Quarkus mentality, but to simply improve the Spring numbers. Quarkus is just a good comparison data point imho.

P.S.S: I'm reporting this here, because the majority of logic relevant to the benchmarks is inside Spring-Framework and not in Spring-Boot.

[dreis2211]

Addendum 1:

There aren't exactly many benchmarks based on Tomcat (the default for SB apps), but comparing the JSON serialization with e.g. a vanilla undertow reveals that there seems to be a quite a lot of framework overhead. And it's not Jackson (because that is used in both undertow and spring (haven't checked Quarkus))

[sdeleuze]

I'm not a particular huge fan of the TechEmpower benchmarks, because they barely reflect real use-cases

@dreis2211 Same, but as I am myself working on some benchmarks to compare Boot 3 JVM and native apps, we can maybe use this opportunity to compare:

• How Spring Framework performs in a default Spring Boot 3 application versus Spring Boot 2 and see what is the evolution if any
• Compare Undertow versus Tomcat on recent versions
• Check if we can find low hanging fruits to optimize

Let's maybe start by focusing on the https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Java/spring. Do you have the bandwidth to create a branch with a Spring Boot 3.0.0-SNAPSHOT (we are close to GA so should be pretty stable) version of it, upgrading to Java 17, Jakarta EE, etc. and compare the result you have locally with the Spring Boot 2.6.9 version used currently? And to compare Undertow versus Tomcat on both?

[dreis2211]

I will have some time next week or earliest this weekend to do some preliminary tests, I guess.

[dreis2211]

Fun fact, apparently the spring benchmarks already run with Undertow and Round 21 doesn't include these changes yet.

[dreis2211]

So, I had a bit of time tonight:

Local specs:
Model: MacBookPro:
Processor: 8-Core Intel Core i9 2,3 GHz
Memory: 32 GB

NOTE: The results should be taken with a grain of salt. After all, local benchmarks are not really isolated and might be influenced by things running on my machine.

Spring-Boot 2.6.9 (Undertow):

Spring-Boot 2.6.9 (Tomcat):

Spring-Boot 3.0.0-SNAPSHOT (Undertow):

Spring-Boot 3.0.0-SNAPSHOT (Undertow) (with produces on GetMapping):

Spring-Boot 3.0.0-SNAPSHOT (Tomcat) (with produces on GetMapping):

Spring-Boot 2.3.1 (Tomcat) (Round 20):

Quarkus:

I'm sort of focussing on the Plaintext & JSON ones here as the others are somewhat volatile and due to the database involvement include some more flakyness not under any frameworks control. What I found was that there seems to be a substantial overhead if the application needs to determine the producible media types. E.g. the changes that moved from 2.3.1 & 2.6.9 include the produces field on the GetMapping annotations for plaintext. As one can see the different runs for 3.0.0 also differ largely for JSON. And there the only change is again the manual produces on the endpoint. Maybe the following plays a role here:

spring-framework/spring-webmvc/src/main/java/org/springframework/web/servlet/mvc/method/annotation/AbstractMessageConverterMethodProcessor.java

Lines 385 to 395 in b72ee5f

	for (HttpMessageConverter<?> converter : this.messageConverters) {
	if (converter instanceof GenericHttpMessageConverter && targetType != null) {
	if (((GenericHttpMessageConverter<?>) converter).canWrite(targetType, valueClass, null)) {
	result.addAll(converter.getSupportedMediaTypes(valueClass));
	}
	}
	else if (converter.canWrite(valueClass, null)) {
	result.addAll(converter.getSupportedMediaTypes(valueClass));
	}
	}
	return (result.isEmpty() ? Collections.singletonList(MediaType.ALL) : new ArrayList<>(result));

.

Funny enough I came across this method already when playing around with the type pollution agent from @franz1981 / RedHat, which yields the following output:

--------------------------
Type Pollution Statistics:
--------------------------
1:      org.springframework.http.converter.json.MappingJackson2HttpMessageConverter
Count:  58366
Types:
        org.springframework.http.converter.HttpMessageConverter
        org.springframework.http.converter.GenericHttpMessageConverter
Traces:
        org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.getProducibleMediaTypes(AbstractMessageConverterMethodProcessor.java:385)
                class: org.springframework.http.converter.HttpMessageConverter
                count: 29208
        org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.getProducibleMediaTypes(AbstractMessageConverterMethodProcessor.java:386)
                class: org.springframework.http.converter.GenericHttpMessageConverter
                count: 29115
        org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.getProducible
MediaTypes(AbstractMessageConverterMethodProcessor.java:387)
                class: org.springframework.http.converter.GenericHttpMessageConverter
                count: 43

If true, this would maybe indicate a scaling/performance problem caused by https://bugs.openjdk.org/browse/JDK-8180450 . Maybe it's not that, but just the fact, that it iterates over 8 message converters for each request to get their producible media types. I haven't really had the time tonight to look deeper into this.

In terms of Tomcat vs Undertow, there seems to be a huge difference for plain text responses (e.g. Tomcat only reaching half of what Undertow achieves).

Cheers,
Christoph

[sdeleuze]

Interesting, better to focus initially on JSON and plaintext indeed.

It would be interesting to do for example a new round of JSON test with only MappingJackson2HttpMessageConverter and remove the other default converters.

I can also provide a branch that removes this instanceof check, I will try to create that tomorrow and share it with you.

[dreis2211]

I think AbstractMessageConverterMethodProcessor.writeWithMessageConverters is indeed something to look at.

E.g. if I force the content-type to be preselected (a.k.a skipping even more logic than just the producible media types) via

	@GetMapping(value = "/plaintext")
	String plaintext(HttpServletResponse response) {
		response.setContentType("text/plain");
		return "Hello, World!";
	}

	@GetMapping(value = "/json")
	Message json(HttpServletResponse response) {
		response.setContentType("application/json");
		return new Message("Hello, World!");
	}

it yields the following numbers:

Spring-Boot 2.6.9 (Undertow):

Remember - the status quo of that was:

[dreis2211]

Another finding: the current tests run inside a Debian Buster image. With Debian Bullseye I can reach a substantially better score on both JSON and Plaintext with 3.0.0 and the forced Content-Type:

[sdeleuze]

@dreis2211 Pretty good results on skipping instanceof GenericHttpMessageConverter checks, see #29438. I was able to confirm the gains on 2 different benchmarks, but if you can check on your side as well, please share the results in the PR.

[GeorgeSalu]

is it worth testing with jetty ?

[dreis2211]

@GeorgeSalu Generally, I wouldn't worry too much about Jetty or Undertow. Tomcat is usually the one that falls behind a bit.

But I've compared it just for you - with latest snapshots, bullseye and enforced content-type (throughout all benchmark modes and thus having a better weighted score overall):

Undertow

Jetty

From personal experience, the margin between them is not as huge as one might read from this results. I'm in fact a little bit surprised about the numbers generally. I have done some memory improvements that don't explain these results, though. I need to take a deeper look what changed in the snapshots in the meanwhile. What has changed is that my Macbook runs on Ventura now. Maybe there have been major improvements to the host OS that generally boost the numbers. Don't forget there is a bit of flakyness involved doing these on a local setup.

Anyhow: Back to your question. Undertow usually wins, also in internal loadtests that we do. There is also a dated benchmark comparison over at https://www.baeldung.com/spring-boot-servlet-containers that hasn't drastically changed in my opinion over recent years.

Cheers,
Christoph

[dreis2211]

I've run a new round of benchmarks to have comparable results between Spring-Boot 3.0.0 RC1 and RC2

RC1

RC2

There is an upgrade from Undertow 2.2.x to 2.3.x - I wouldn't be surprised if this brings some improvements. Next to several tiny improvements in Spring itself.