performance regression when running admin with jetty 9

[brharrington]

In karyon 2.3.0 we are seeing high CPU load due to the admin. The problem goes away with karyon 2.4.0 so I suspect the issue is something with the upgrade to jetty 9. I don't know what the future plans are regarding the jetty version, so this is mostly an FYI for when you plan to move forward again.

Details

After running for a while, it would sometimes take an hour or so, we see a big jump in CPU usage:

Jetty threads are using the most CPU:

Results of profiling show:

The jars for karyon/jetty were:

• karyon2-admin-2.3.0.jar
• karyon2-admin-eureka-plugin-2.3.0.jar
• karyon2-admin-web-2.3.0.jar
• jetty-http-9.3.0.M1.jar
• jetty-io-9.3.0.M1.jar
• jetty-security-9.3.0.M1.jar
• jetty-server-9.3.0.M1.jar
• jetty-servlet-9.3.0.M1.jar
• jetty-util-9.3.0.M1.jar

[tbak]

I have observed the same issue + leaking TCP connections in CLOSE_WAIT state. I found two reported issues matching this behavior:

• https://bugs.eclipse.org/bugs/show_bug.cgi?id=444031
• http://dev.eclipse.org/mhonarc/lists/jetty-users/msg04665.html.