Connection reset on HTTP/2 when using CompressionHandler with default WindowRateControl

[ivosek]

Jetty version(s)
Jetty 12.1.3 embedded

Jetty Environment
ee10, ee11 (ee8 and ee9 are unaffected)

HTTP version
HTTP 2

Java version/vendor

openjdk version "21.0.5" 2024-10-15 LTS
OpenJDK Runtime Environment Temurin-21.0.5+11 (build 21.0.5+11-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.5+11 (build 21.0.5+11-LTS, mixed mode, sharing)

OS type/version
Ubuntu 24.04.3 LTS

Description

We tried to migrate from Jetty 12.0.29 to 12.1.3 and when CompressionHandler is used, requests starts to break due to connection resets.

Problem is not present:

• on Jetty 12.0.29 with GzipHandler and HTTP/2
• on Jetty 12.1.3 with CompressionHandler and HTTP 1.1
• on Jetty 12.1.3 without CompressionHandler and with HTTP/2

I have also tried it with the fix from #13812 and it still fails.

It fails every time when it serves development version of our frontend (huge amount of .dcc.js files) but with production release it seems to be working well.

When we were trying to create minimal reproducion sample we found out it works well when we added:

h2.setRateControlFactory(new WindowRateControl.Factory(1024));

(We added this line just by luck because @lorban suggested this line while we were preparing minimal reproduction sample for another issue half year ago - see #12776)

So this might not be a bug but feature.

In org.eclipse.jetty.http2.server.AbstractHTTP2ServerConnectionFactory I can see that default value is new WindowRateControl.Factory(128).

My questions:

• Isn't this default value set too low, when single web browser client session can hit this limitation?
• Why this happens only when CompressionHandler is used? Even when we tried to change this value to WindowRateControl.Factory(4) it never fails when CompressionHandler is not used.
• Why this does not occurs on Jetty 12.0.x when there is also default value WindowRateControl.Factory(128)?

How to reproduce?

reproduction.zip

1. Run:

mvn compile exec:java

2. Open web browser (Chromium based or Firefox) and open: https://localhost:9443/

[lorban]

The problem seems to be caused some form of expectation mismatch between the Core and Servlet APIs. Basically, both the CompressionHandler and the ServletHandler believe they are responsible for writing the last buffer to the response, so sometimes they both do which makes the GzipEncoderSink throw IllegalStateException("Already released"). When that happens and the protocol is HTTP/2, this triggers a reset which is accounted for by the WindowRateControl so depending on your luck, you may overshoot your event rate.

So the problem is not strictly related to the compression handler, cannot happen with HTTP/1 as there is no reset mechanism and the problem happens only after the last bytes were written (so the client is unaware of the problem) and Jetty 12.0's compression works differently and doesn't seem to be impacted by that problem.

Here are the stack traces I managed to capture of the two last-writes:

java.lang.Throwable: stack of first last-write
	at org.eclipse.jetty.compression.EncoderSink.write(EncoderSink.java:36)
	at org.eclipse.jetty.compression.server.internal.CompressionResponse.write(CompressionResponse.java:147)
	at org.eclipse.jetty.compression.server.internal.CompressionResponse.write(CompressionResponse.java:145)
	at org.eclipse.jetty.server.Response$Wrapper.write(Response.java:836)
	at org.eclipse.jetty.server.handler.ContextResponse.write(ContextResponse.java:56)
	at org.eclipse.jetty.ee10.servlet.ServletContextResponse.write(ServletContextResponse.java:224)
	at org.eclipse.jetty.io.IOResources$PathToSinkCopier.process(IOResources.java:340)
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:377)
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:354)
	at org.eclipse.jetty.io.IOResources.copy(IOResources.java:244)
	at org.eclipse.jetty.http.content.ResourceHttpContent.writeTo(ResourceHttpContent.java:125)
	at org.eclipse.jetty.http.content.HttpContent$Wrapper.writeTo(HttpContent.java:248)
	at org.eclipse.jetty.server.ResourceService.sendData(ResourceService.java:681)
	at org.eclipse.jetty.server.ResourceService.doGet(ResourceService.java:207)
	at org.eclipse.jetty.ee10.servlet.ResourceServlet.doGet(ResourceServlet.java:593)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)
	at org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:752)
	at org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1620)
	at org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1554)
	at org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:807)
	at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:442)
	... 11 more

java.lang.Throwable: stack of second last-write
	at org.eclipse.jetty.compression.EncoderSink.write(EncoderSink.java:36)
	at org.eclipse.jetty.compression.server.internal.CompressionResponse.write(CompressionResponse.java:147)
	at org.eclipse.jetty.server.Response$Wrapper.write(Response.java:836)
	at org.eclipse.jetty.server.handler.ContextResponse.write(ContextResponse.java:56)
	at org.eclipse.jetty.ee10.servlet.ServletContextResponse.write(ServletContextResponse.java:224)
	at org.eclipse.jetty.ee10.servlet.HttpOutput.channelWrite(HttpOutput.java:255)
	at org.eclipse.jetty.ee10.servlet.HttpOutput.complete(HttpOutput.java:502)
	at org.eclipse.jetty.ee10.servlet.ServletContextResponse.completeOutput(ServletContextResponse.java:212)
	at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:587)
	at org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:469)
	at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1220)
	at org.eclipse.jetty.compression.server.CompressionHandler.handle(CompressionHandler.java:346)
	at org.eclipse.jetty.server.Server.handle(Server.java:195)
	at org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:680)
	at org.eclipse.jetty.util.thread.Invocable$ReadyTask.run(Invocable.java:178)
	at org.eclipse.jetty.http2.server.internal.HttpStreamOverHTTP2$1.run(HttpStreamOverHTTP2.java:146)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:1009)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1239)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1194)
	at java.base/java.lang.Thread.run(Thread.java:1474)

In the 1st stack, IOResources.copy() is responsible for doing the last write. In the 2nd stack, HttpOutput.complete() decides that the empty buffer should be used for doing again the last write.

I've yet to figure out why this problem doesn't happen for all requests, there seems to be a race condition somewhere making this happen at random.

Thanks for the report! We're going to work on a fix ASAP.

[lorban]

@ivosek we've identified the bug: it's mismatch between the expectations of the Core API and the ee10/11 servlet state machines. The failure always happens but too late in HTTP/1.1 to have any noticeable effect, and only has a visible effect in HTTP/2 when the flow control window delays the response so the reset frame has time to reach the client.

We are considering two different ways to fix this problem, but #13914 has our preference for now.

Would you mind locally testing that branch?

Thanks!