NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpFields$Mutable.contains(String)" because "this._fields" is null

[thomas-k-git]

Under contention, when using jetty + jersey and having a resource configured with org.glassfish.jersey.server.model.ResourceMethod.Builder#managedAsync and an EOF exception mapping filter and closing many client connections this NPE can be observed regularly.

Per analysis from the jetty team, the response object seems to wrongly re-used after completion (not in line with servlet spec) of the request triggering this NPE.

version(s)
jetty 12.0.19 + jersey-bom 2.46 (same behavior in 2.34 as in attached reproducer project)

Jetty Environment
ee8

Java version/vendor (use: java -version)
openjdk 21.0.7 2025-04-15 LTS
OpenJDK Runtime Environment Temurin-21.0.7+6 (build 21.0.7+6-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.7+6 (build 21.0.7+6-LTS, mixed mode, sharing)

OS type/version
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.5 LTS
Release: 22.04
Codename: jammy

Description
2025-04-30 13:36:16 GRAVE org.glassfish.jersey.server.ServerRuntime$Responder writeResponse - Error while closing the output stream in order to commit response. java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpFields$Mutable.contains(String)" because "this._fields" is null at org.eclipse.jetty.ee8.nested.Response.containsHeader(Response.java:326) at org.glassfish.jersey.servlet.internal.ResponseWriter.writeResponseStatusAndHeaders(ResponseWriter.java:135) at org.glassfish.jersey.server.ServerRuntime$Responder$1.getOutputStream(ServerRuntime.java:625) at org.glassfish.jersey.message.internal.CommittingOutputStream.commitStream(CommittingOutputStream.java:171) at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:276) at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:232) at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:247) at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:842) at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:389) at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:707) at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:373) at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:419) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$4.run(ServerRuntime.java:872) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:244) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:889) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:867) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2$1.run(ServerRuntime.java:821) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:244) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2.run(ServerRuntime.java:811) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583)

in specific setup with async thread pool, jersey, and EOFMapper

How to reproduce?
see attached project. running it a few times prodcues the NPE for me, sometimes also just issue 2.

npe-report.zip

Also seen issue 2
java.lang.IllegalStateException: AsyncContext completed and/or Request lifecycle recycled
at org.eclipse.jetty.ee8.nested.AsyncContextState.state(AsyncContextState.java:42)

Also seen issue 3
java.io.IOException: content-length 11 != 0 written
at org.eclipse.jetty.server.internal.HttpChannelState$ChannelCallback.succeeded(HttpChannelState.java:1548)

** Also seen*
In this similar setup in our more complex application we also saw threads of the secondary thread pool (in the repro called "CUSTOM") hanging forever, that we think is related, but will try to get a separate reproducer for that. But we think it might be related / caused originally by this NPE

see attached files.

npe-report.zip

originally reported at jetty in this ticket, but redirected here

[senivam]

as I'm looking into it, Jersey 2.x is not declared to support Jetty 12.x. The version of Jetty which is used in the Jersey 2.x is 9.4.55.v20240627. When using this version the exception does not occur. The latest Jetty 9.x version is 9.4.57.v20241219 in which the exception does not occur as well. Why not use Jetty 9.x in conjunction with Jersey 2.x?

[thomas-k-git]

@senivam I tried to look for it, but couldn't really find it, where is the official info on which jersey goes with which jetty?

regarding jetty 9, it's EOL: jetty/jetty.project#7958
same goes for jetty 10, which is actually the reason why we are in the process of migrating to jetty12

for context: the attached project is a minimal reproducer for a problem that we have in our huge production system, which is complicated to migrate to other versions anyways, so would prefer moving forwards recent jetty in the long run.

[senivam]

OK, I've investigated deeper, and it looks, the server just behaves accordingly to the reproducer. In my investigation, the reproducer actually does not work on the 9.4.x Jetty either. It does not visibly throw any exception because it's hidden behind the

catch (InterruptedException e) {
			throw new RuntimeException(e);
		} 

otherwise it would fail either. The client sends the request using

HttpRequest request = HttpRequest.newBuilder()
				.POST(HttpRequest.BodyPublishers.ofInputStream(HangInputStream::new))
				.uri(URI.create("http://localhost:" + PORT + "/"))
				.timeout(Duration.ofDays(1))
				.build();

and the HangInputStream structure is

static class HangInputStream extends InputStream {

		@Override
		public int read() {
			try {
				Thread.sleep(Long.MAX_VALUE);
				return -1;
			} catch (InterruptedException e) {
				throw new RuntimeException(e);
			}
		}
	}

So, the server attempts to read from the stream and drops into sleep for the Long.MAX_VALUE while in the main thread the
httpClient.shutdownNow(); httpClient.close(); is called. There is no way the server would wake before those shutdown/close methods are called on the client. That is why exception is thrown - really the thread is closed. Having this reproducer I do not see how the server could behave any better.

[thomas-k-git]

Agreed, the reproducer is set up to simulate a EOF/connection close from the client, while a server thread is reading it (through the network socket).

The closing of the connection by the client triggers the JettyEofExceptionMapper trying to write to the response.

However this only triggers the actual problem (from how I understand the jetty team's response) that then jersey in multiple threads accesses the same reponse object without the required coordination - or completes the request and then access it again (eg reponse writing task for the EOF mapper?) - which gives the NPE because the HTTP reponse is being re-used for the next unrelated request in the queue.

So the way I read it, the NPE is only a symptom from missing synchronization (or order) in accessing org.eclipse.jetty.ee8.nested.Response objects in this situation.

Sorry that I cannot be more specific / have more details, I lack enough understanding of the jersey async task framework here I guess to put this into the right context

[senivam]

So, having your explanation, I've figured out, that prior to Jetty 12.0.19 (like 12.0.18 and below) the exception is:

Threw IOException with message: shutdownNow
2025-05-07 15:22:07 SEVERE org.glassfish.jersey.server.ServerRuntime$Responder writeResponse - Error while closing the output stream in order to commit response.
java.lang.UnsupportedOperationException: Read Only
	at org.eclipse.jetty.server.internal.ResponseHttpFields$2.set(ResponseHttpFields.java:227)
	at org.eclipse.jetty.server.internal.ResponseHttpFields$2.set(ResponseHttpFields.java:169)
	at org.eclipse.jetty.http.HttpFields$Mutable.put(HttpFields.java:1233)
	at org.eclipse.jetty.http.HttpFields$Mutable.put(HttpFields.java:1361)
	at org.eclipse.jetty.ee8.nested.Response.setContentLength(Response.java:800)
	at org.glassfish.jersey.servlet.internal.ResponseWriter.writeResponseStatusAndHeaders(ResponseWriter.java:123)
	at org.glassfish.jersey.server.ServerRuntime$Responder$1.getOutputStream(ServerRuntime.java:639)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.commitStream(CommittingOutputStream.java:174)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:279)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:235)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:250)
	at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:568)
	at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:406)
	at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:721)
	at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:380)
	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:426)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$4.run(ServerRuntime.java:886)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:903)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:881)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2$1.run(ServerRuntime.java:835)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2.run(ServerRuntime.java:825)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

and after - 12.0.19 (+) the exception is:

Threw IOException with message: shutdownNow
2025-05-07 15:24:51 SEVERE org.glassfish.jersey.server.ServerRuntime$Responder writeResponse - Error while closing the output stream in order to commit response.
java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpFields$Mutable.put(org.eclipse.jetty.http.HttpHeader, long)" because "this._fields" is null
	at org.eclipse.jetty.ee8.nested.Response.setContentLength(Response.java:800)
	at org.glassfish.jersey.servlet.internal.ResponseWriter.writeResponseStatusAndHeaders(ResponseWriter.java:123)
	at org.glassfish.jersey.server.ServerRuntime$Responder$1.getOutputStream(ServerRuntime.java:639)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.commitStream(CommittingOutputStream.java:174)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:279)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:235)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:250)
	at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:568)
	at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:406)
	at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:721)
	at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:380)
	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:426)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$4.run(ServerRuntime.java:886)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:903)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:881)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2$1.run(ServerRuntime.java:835)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2.run(ServerRuntime.java:825)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Jersey is 2.46 all the way. Is there any exception you are expecting here or it should work as in Jetty 9.4.x without throwing any exception?

[thomas-k-git]

I think any of the two exceptions is just a symptom of what joakime mentioned here, where in this jersey situation the servlet spec is violated I presume.

So when jersey calls Response.setContentLength (on jetty), I guess the section "5.8 Lifetime of the Response Object" of the spec applies. I'm assuming those parts:

If asynchronous processing
on the associated request is started, then the response object remains valid until
complete method on AsyncContext is called. Containers commonly recycle response
objects in order to avoid the performance overhead of response object creation. The
developer must be aware that maintaining references to response objects for which
startAsync on the corresponding request has not been called, outside the scope
described above may lead to non-deterministic behavior.

in the reproducer, we have started an AsyncContext, via org.glassfish.jersey.server.model.ResourceMethod.Builder#managedAsync - so this section applies.

I think the complete is called already on the AsyncContext (or it's response) but we are still writing it, leading to the non-deterministic behavior = NPE or UnsupportedOperationException.

So I would guess to comply with the spec, jersey would need to ensure that Response.setContentLength is not called at all if already completed.

[thomas-k-git]

Or in other words, the reason for this could be a race condition between multiple concurrent calls to the methods mentioned servlet spec 5.7: sendError / complete / setContentLength / ...
(but I haven't analyzed the code deeply enough to know)

[senivam]

the issue is that Jersey is not trying to set content legth on some unknown response. It actually processes reponce provided by the JettyEofExceptionMapper which is

return Response.status((Response.StatusType) Status.BAD_REQUEST)
				.type(ExceptionMapperUtil.APPLICATION_JSON_UTF_8)
				.entity("Status: 429").build();

thus it tries to set content-legth of "Status: 429".length() which is 11 (greater than 0) and the whole situation occurs.

[thomas-k-git]

Ah! That's where the 11 comes from in

issue 3
java.io.IOException: content-length 11 != 0 written
at org.eclipse.jetty.server.internal.HttpChannelState$ChannelCallback.succeeded(HttpChannelState.java:1548)

[senivam]

As I'm seeing this from Jersey's point of view, it's not failure of asynchronous request processing. It's one single request / response sequence which does not handle errors properly, but it looks like that this case is on Jetty's side. Jersey just tries to use API provided by Jetty. But as I was saying from the very beginning - the 2.x version of Jersey does not support Jetty 12.0 yet. And this issue shows that it would not be an easy task to upgrade Jersey to support the latest Jetty.

[thomas-k-git]

Just to avoid having unsupported combos, where should we be looking to stay within supported combinations? In the areas of the user guide I'm only seeing JDK compabilities.

(as we were about to release actually in this combination)

[senivam]

because it's only in pom.xml of the project. We were not aware of the Jetty's 9.4 EOL.

the 2.x works on the 9.4.x Jetty,
the 3.0 - on the 11.0 Jetty's version,
and the 3.1 starts to support 12.0 Jetty but for Jakarta EE 10 - this means jakarta namespace instead of javax .

If the 12.0 Jetty is meant to be supporting all Jakarta's versions and other Jettys are after the EOL, we should consider updating to it in all versions of Jersey (instead of using 9.4.x and 11.0). But currently the status is that.

if you are migrating to Jetty 12.0, consider migrating to Jersey 3.1 also, this is the most current version of Jersey we can provide.

[thomas-k-git]

Thanks for that info and input!
Internally we'll be going into discussions immediately about jersey 3.1 updating, though we were hoping to do one after another for splitting the risk of changing two moving parts.
Can you say if 2.x will be getting jetty12 support at some point? If not, we'll likely plan to move forward together with both at the same time.