Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IllegalStateException when stopping Server with pending requests #11016

Closed
sbordet opened this issue Dec 3, 2023 · 2 comments · Fixed by #11017
Closed

IllegalStateException when stopping Server with pending requests #11016

sbordet opened this issue Dec 3, 2023 · 2 comments · Fixed by #11017
Assignees
@sbordet
Labels
Bug For general bugs on Jetty side

Comments

@sbordet
Copy link
Contributor

sbordet commented Dec 3, 2023

Jetty version(s)
12+

Jetty Environment
ee10

Description
Stopping a Server when a request is suspended, for example in case of a "long-polling" request yields:

java.lang.IllegalStateException: s=HANDLING rs=COMPLETED os=ABORTED is=IDLE awp=false se=false i=false al=0
	at org.eclipse.jetty.ee10.servlet@12.0.4-SNAPSHOT/org.eclipse.jetty.ee10.servlet.ServletChannelState.recycle(ServletChannelState.java:1086)
	at org.eclipse.jetty.ee10.servlet@12.0.4-SNAPSHOT/org.eclipse.jetty.ee10.servlet.ServletChannel.recycle(ServletChannel.java:418)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.internal.CompletionStreamWrapper.onCompletion(CompletionStreamWrapper.java:74)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.internal.CompletionStreamWrapper.failed(CompletionStreamWrapper.java:53)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.completeStream(HttpChannelState.java:704)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.failed(HttpChannelState.java:673)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.internal.HttpChannelState$ErrorCallback.failed(HttpChannelState.java:1636)
	at org.eclipse.jetty.util@12.0.4-SNAPSHOT/org.eclipse.jetty.util.Callback$Nested.failed(Callback.java:429)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.internal.HttpChannelState$ChannelResponse.lambda$failed$2(HttpChannelState.java:1241)
	at org.eclipse.jetty.util@12.0.4-SNAPSHOT/org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:191)
	at org.eclipse.jetty.util@12.0.4-SNAPSHOT/org.eclipse.jetty.util.thread.SerializedInvoker.run(SerializedInvoker.java:117)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.internal.HttpChannelState$ChannelResponse.failed(HttpChannelState.java:1241)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.internal.HttpConnection$SendCallback.reset(HttpConnection.java:759)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.internal.HttpConnection$HttpStreamOverHTTP1.send(HttpConnection.java:1421)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.HttpStream$Wrapper.send(HttpStream.java:179)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.internal.HttpChannelState$ChannelResponse.write(HttpChannelState.java:1181)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.Response$Wrapper.write(Response.java:693)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.handler.ContextResponse.write(ContextResponse.java:56)
	at org.eclipse.jetty.ee10.servlet@12.0.4-SNAPSHOT/org.eclipse.jetty.ee10.servlet.HttpOutput.channelWrite(HttpOutput.java:204)
	at org.eclipse.jetty.ee10.servlet@12.0.4-SNAPSHOT/org.eclipse.jetty.ee10.servlet.HttpOutput.complete(HttpOutput.java:435)
	at org.eclipse.jetty.ee10.servlet@12.0.4-SNAPSHOT/org.eclipse.jetty.ee10.servlet.ServletContextResponse.completeOutput(ServletContextResponse.java:209)
	at org.eclipse.jetty.ee10.servlet@12.0.4-SNAPSHOT/org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:588)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1202)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1195)
	at org.eclipse.jetty.ee10.servlet@12.0.4-SNAPSHOT/org.eclipse.jetty.ee10.servlet.ServletChannelState.runInContext(ServletChannelState.java:1213)
	at org.eclipse.jetty.ee10.servlet@12.0.4-SNAPSHOT/org.eclipse.jetty.ee10.servlet.ServletChannelState.complete(ServletChannelState.java:743)
	at org.eclipse.jetty.ee10.servlet@12.0.4-SNAPSHOT/org.eclipse.jetty.ee10.servlet.AsyncContextState.complete(AsyncContextState.java:61)
	at org.eclipse.jetty.ee10.servlet@12.0.4-SNAPSHOT/org.eclipse.jetty.ee10.servlet.AsyncServletLongPollTest$2.destroy(AsyncServletLongPollTest.java:193)
	at org.eclipse.jetty.ee10.servlet@12.0.4-SNAPSHOT/org.eclipse.jetty.ee10.servlet.ServletHolder.destroyInstance(ServletHolder.java:464)
	at org.eclipse.jetty.ee10.servlet@12.0.4-SNAPSHOT/org.eclipse.jetty.ee10.servlet.ServletHolder.doStop(ServletHolder.java:441)
	at org.eclipse.jetty.util@12.0.4-SNAPSHOT/org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
	at org.eclipse.jetty.ee10.servlet@12.0.4-SNAPSHOT/org.eclipse.jetty.ee10.servlet.ServletHandler.doStop(ServletHandler.java:298)
	at org.eclipse.jetty.util@12.0.4-SNAPSHOT/org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
	at org.eclipse.jetty.util@12.0.4-SNAPSHOT/org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
	at org.eclipse.jetty.util@12.0.4-SNAPSHOT/org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.Handler$Abstract.doStop(Handler.java:476)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.handler.ContextHandler.lambda$doStop$1(ContextHandler.java:686)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.call(ContextHandler.java:1141)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:686)
	at org.eclipse.jetty.ee10.servlet@12.0.4-SNAPSHOT/org.eclipse.jetty.ee10.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:1324)
	at org.eclipse.jetty.ee10.servlet@12.0.4-SNAPSHOT/org.eclipse.jetty.ee10.servlet.ServletContextHandler.doStop(ServletContextHandler.java:1075)
	at org.eclipse.jetty.util@12.0.4-SNAPSHOT/org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
	at org.eclipse.jetty.util@12.0.4-SNAPSHOT/org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
	at org.eclipse.jetty.util@12.0.4-SNAPSHOT/org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.Handler$Abstract.doStop(Handler.java:476)
	at org.eclipse.jetty.server@12.0.4-SNAPSHOT/org.eclipse.jetty.server.Server.doStop(Server.java:669)
@sbordet sbordet added the Bug For general bugs on Jetty side label Dec 3, 2023
@sbordet sbordet self-assigned this Dec 3, 2023
@sbordet
Copy link
Contributor Author

sbordet commented Dec 3, 2023

The Server stop results in the application to try to complete the suspended request via asyncContext.complete().

This re-enters the ServletChannel state machine, but tries to complete the Handler callback directly, which eventually calls recycle(), which checks that ServletChannel._state != HANDLING, failing with the IllegalStateException.

Seems wrong to me to try to fail the Handler callback from any state: it should only be failed in the TERMINATED state, which calls ServletChannel.onCompleted()` (like it is done when the callback is succeeded).

sbordet added a commit that referenced this issue Dec 3, 2023
@sbordet
Fixes #11016 - Jetty 12 IllegalStateException when stopping Server wi…
dae910a 
…th pending requests

* Made ServletChannel error handling more robust.
A failure in error handling is now remembered so that the Handler callback can be failed later.
* Avoid failing the Handler callback from ServletChannel.abort(), as it is too early: should be failed when processing the TERMINATED state, similarly to when it is succeeded.
* Removed dead code from HttpConnection.SendCallback.reset(), since response is always non-null.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
@sbordet sbordet linked a pull request Dec 3, 2023 that will close this issue
Fixes #11016 - Jetty 12 IllegalStateException when stopping Server wi… #11017
Merged
@sbordet sbordet moved this to 🏗 In progress in Jetty 12.0.4 - FROZEN Dec 3, 2023
@lorban lorban mentioned this issue Dec 4, 2023
Closed
@gregw gregw moved this to 🏗 In progress in Jetty 12.0.5 - FROZEN Dec 13, 2023
sbordet added a commit that referenced this issue Dec 15, 2023
@sbordet
Fixes #11016 - Jetty 12 IllegalStateException when stopping Server wi… (
eb1e9eb 
#11017)

* Made ServletChannel error handling more robust.
A failure in error handling is now remembered so that the Handler callback can be failed later.
* Avoid failing the Handler callback from ServletChannel.abort(), as it is too early: should be failed when processing the TERMINATED state, similarly to when it is succeeded.
* Removed dead code from HttpConnection.SendCallback.reset(), since response is always non-null.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>
@sbordet
Copy link
Contributor Author

sbordet commented Dec 15, 2023

Fixed by #11017.

@sbordet sbordet closed this as completed Dec 15, 2023
@github-project-automation github-project-automation bot moved this from 🏗 In progress to ✅ Done in Jetty 12.0.5 - FROZEN Dec 15, 2023
@sbordet sbordet mentioned this issue Dec 17, 2023
Open
@joakime joakime changed the title Jetty 12 IllegalStateException when stopping Server with pending requests IllegalStateException when stopping Server with pending requests Dec 18, 2023
@lorban lorban mentioned this issue Apr 2, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sbordet sbordet

Labels
Bug For general bugs on Jetty side
Projects
No open projects
Jetty 12.0.5 - FROZEN
Status: ✅ Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixes #11016 - Jetty 12 IllegalStateException when stopping Server wi… jetty/jetty.project
1 participant
@sbordet