java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpFields$Mutable.contains(String)" because "this._fields" is null

[thomas-k-git]

Jetty version(s)
12.0.19

Jetty Environment
ee8

Java version/vendor (use: java -version)
openjdk 21.0.7 2025-04-15 LTS
OpenJDK Runtime Environment Temurin-21.0.7+6 (build 21.0.7+6-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.7+6 (build 21.0.7+6-LTS, mixed mode, sharing)

OS type/version
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.5 LTS
Release: 22.04
Codename: jammy

Description
2025-04-30 13:36:16 GRAVE org.glassfish.jersey.server.ServerRuntime$Responder writeResponse - Error while closing the output stream in order to commit response. java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpFields$Mutable.contains(String)" because "this._fields" is null at org.eclipse.jetty.ee8.nested.Response.containsHeader(Response.java:326) at org.glassfish.jersey.servlet.internal.ResponseWriter.writeResponseStatusAndHeaders(ResponseWriter.java:135) at org.glassfish.jersey.server.ServerRuntime$Responder$1.getOutputStream(ServerRuntime.java:625) at org.glassfish.jersey.message.internal.CommittingOutputStream.commitStream(CommittingOutputStream.java:171) at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:276) at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:232) at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:247) at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:842) at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:389) at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:707) at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:373) at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:419) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$4.run(ServerRuntime.java:872) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:244) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:889) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:867) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2$1.run(ServerRuntime.java:821) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:244) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2.run(ServerRuntime.java:811) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583)

in specific setup with async thread pool, jersey, and EOFMapper

How to reproduce?
see attached project. running it a few times prodcues the NPE for me, sometimes also just issue 2.

Also seen issue 2
java.lang.IllegalStateException: AsyncContext completed and/or Request lifecycle recycled
at org.eclipse.jetty.ee8.nested.AsyncContextState.state(AsyncContextState.java:42)

Also seen issue 3
java.io.IOException: content-length 11 != 0 written
at org.eclipse.jetty.server.internal.HttpChannelState$ChannelCallback.succeeded(HttpChannelState.java:1548)

** Also seen*
In this similar setup in our more complex application we also saw threads of the secondary thread pool (in the repro called "CUSTOM") hanging forever, that we think is related, but will try to get a separate reproducer for that. But we think it might be related / caused originally by this NPE

see attached files.

npe-report.zip

[thomas-k-git]

full stack traces in the files in the attached zip

[joakime]

How to reproduce?
see attached project. running it a few times prodcues the NPE for me, sometimes also just issue 2.

You are attempting to use the Request after it has been completely finished.
You are actually accessing a Request object that has been recycled.

Also seen issue 2 java.lang.IllegalStateException: AsyncContext completed and/or Request lifecycle recycled at org.eclipse.jetty.ee8.nested.AsyncContextState.state(AsyncContextState.java:42)

You are violating the AsyncContext rules and are accessing the AsyncContext after the Request has fully completed.

Also seen issue 3 java.io.IOException: content-length 11 != 0 written at org.eclipse.jetty.server.internal.HttpChannelState$ChannelCallback.succeeded(HttpChannelState.java:1548)

You have an HttpServletResponse with a stated Content-Length: 11 in a response header set, but your servlet didn't write the full 11 bytes, in fact it wrote 0. This error is produced when the Request/Response exchange has completed.

[joakime]

java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpFields$Mutable.contains(String)" because "this._fields" is null
in specific setup with async thread pool, jersey, and EOFMapper

This is also a tell-tale sign that you are using a request (or response) object after it has been completed and recycled.

[thomas-k-git]

@joakime thanks for the hints.
I guess this is then more a ticket for the jersey project.
I don't have too deep insight into the jetty project, but would there be more useful information to send along to the jersey bug ticket?

To confirm my understanding:

• With recycling, do you mean calls on the HttpChannel#recycle (and then the Request / Response#recycle)?
• With complete you mean javax.servlet.AsyncContext#complete ?
• org.eclipse.jetty.ee8.nested.Response is intentionally not thread safe and safety needs to be guaranteed by user.
• and due to this you would probably not prevent this NPE if coming from that?

[joakime]

To confirm my understanding:

• With recycling, do you mean calls on the HttpChannel#recycle (and then the Request / Response#recycle)?

Many large / complex objects in EE are recycled.
The HttpServletRequest is a prime example.
The Servlet spec allows for request recycling, and some containers (like ours) do that.

• With complete you mean javax.servlet.AsyncContext#complete ?

I mean complete.
The AsyncContext.complete is a method you call.
The best (but not perfect) place to catch AsyncContext completion is probably AsyncListener.onComplete(), but that's subject to the typical race conditions in threading.

• org.eclipse.jetty.ee8.nested.Response is intentionally not thread safe and safety needs to be guaranteed by user.

The response isn't recycled, but it is reset and rendered useless once the Request / Response exchange completes.

• and due to this you would probably not prevent this NPE if coming from that?

It is very possible to write sane async / multi-threaded servlet code.
Many projects do that just fine.

The more complex a project becomes the more likely it seem to trip over servlet things like this.
Jersey isn't alone in this regard.

This was filed against Spring ...

• Thread-unsafe usage of HttpServletResponse corrupting jetty HttpFields state in reactive ResponseBodyEmitter handlers spring-projects/spring-security#9175

But since that issue was filed, many comments in that issue have shown that the issue is present in other containers too, not just Jetty.

[thomas-k-git]

Thanks! For reference I just opened eclipse-ee4j/jersey#5908 with basically this ticket's content.

[janbartel]

This issue appears to be addressed by the jersey project, so closing here also.