Numerous stack traces logged at warning level when running under HTTP/2 (regression in 12.0.15) #12520
Comments
|
@basil thanks for this report and the excellent bisection down to a specific commit. I've not yet tried your reproduction, but am just initially trying to get a general feel for the issue. The stacks indicate that we are trying to normally complete a response that has already been aborted. Since you have an easy reproduction, any chance that you can run this with debug logging turned on for the thanks |
|
@gregw I ran through the scenario described above again, this time with It took me almost a minute of clicking around in the Jenkins UI before the This file is 16 MB (!) in size, covering about 1 minute of Jenkins usage. The |
|
@basil Thanks for that log. The actual problem appears to start at 6:54:19 when I can see a write it failed with a reset connection. This them results in an other exception regarding insufficient content written, which is the cause of the abort, which in turn results in the logged exception about the channel being in ABORT state. Failing gracefully when having a reset slammed in our faces whilst writing a response can be difficult, at least not without accidentally suppressing other exceptions. So we will need to check if a) we can avoid the insufficient content written exception, as that is a result of the resets; and b) should we not log at warning when a response has been aborted before completion? stand by.... |
Signed-off-by: Ludovic Orban <lorban@bitronix.be>
Signed-off-by: Ludovic Orban <lorban@bitronix.be>
Signed-off-by: Ludovic Orban <lorban@bitronix.be>
Signed-off-by: Ludovic Orban <lorban@bitronix.be>
|
@basil I've created a pull request that contains a fix for this bug. While I'm busy ironing out the last details, I'd be glad if you could give it a try to see if it works for you too as I haven't managed to reproduce the problem with a Jenkins build using the Jetty build of that PR. Thanks! |
…11210) Signed-off-by: Ludovic Orban <lorban@bitronix.be>
github-project-automation
bot
moved this from 🏗 In progress
to ✅ Done
in Jetty 12.0.16 FROZEN
Jetty version(s)
12.0.15
Jetty Environment
EE 9
Java version/vendor
OS type/version
Ubuntu 22.04.5 LTS x86_64
Context
jenkinsci/jenkins#9951
Steps to reproduce
Warning
My apologies that these steps are not very minimal. Unfortunately, I could not find a minimal reproducible example outside of Jenkins.
Start by pulling down jenkinsci/jenkins#9951. This does nothing other than pull in jenkinsci/winstone#419, which in turn does nothing other than update Jetty from 12.0.14 to 12.0.15.
Compile it with Java 17 by running
mvn clean verify -DskipTests.Run it as follows:
Once Jenkins comes up, visit https://127.0.0.1:8000 in Firefox and enter the password that was given to you on the console. Click on Install suggested plugins, wait for that to complete, click Skip and continue as admin, click Save and finish, and click Start using Jenkins.
Warning
This appears to be some type of race condition, as I could not find a deterministic way to trigger the issue. However, after doing a few generic actions in the Jenkins UI, I was able to consistently see stack traces within about a minute of clicking around in Firefox.
Do some generic actions in the Jenkins UI. For example, click on New Item, enter a name, select Freestyle Project, click Add Build Step, click Execute Shell, type
echo hello, save the project, and build the project a few times.Check the console where you started Jenkins. You should see the stack traces mentioned below.
If not, open a new tab in Firefox (leaving the old one untouched) and continue clicking around in the UI. Click to go back to the Dashboard, click on the Freestyle project again, build it a few more times, etc. Within a minute, I am always able to see the stack traces mentioned below.
If this bug is being really stubborn, kill the Java process and start Jenkins again, log in again, and browse around again in a new tab. I am always able to reproduce the issue within about a minute after doing this.
Expected results
Note
These are the actual results with Jenkins at tip-of-trunk, running Jetty 12.0.14.
Back in the console where you started Jenkins, the logs should be clean after:
Actual results
The console is full of warnings and stack traces like this:
Evaluation
Note
The warnings are not related to gzip compression, as they appear whether or not
GzipHandleris enabled. In the above example,GzipHandleris enabled (and part of the stack trace), but it can be disabled by starting Jenkins with--compression=none, in which case the problem still appears with a slightly less deep stack trace that does not includeGzipHandler.I bisected this change in behavior to commit f48be0d by @sbordet, part of #12313 and #12370.
I am not comfortable delivering Jetty 12.0.15 to Jenkins users with these stack traces being logged at warning level; in other words, this issue is blocking the Jenkins project from upgrading to the latest release of Jetty.
The text was updated successfully, but these errors were encountered: