Could not send response error 500

[rocky-peng]

I have a stress test applied to my web app running in jetty container,the jetty throws some exceptions like follow:

2016-10-27 16:29:20.532:WARN:oejs.HttpChannel:qtp314357161-104: Could not send response error 500: java.io.IOException: org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException
2016-10-27 16:29:20.682:WARN:oejs.ServletHandler:qtp314357161-245: /api3/new_square/recommend
java.io.IOException: org.eclipse.jetty.util.SharedBlockingCallback$BlockerTimeoutException
        at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:234)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:141)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:383)
        at java.io.OutputStream.write(OutputStream.java:75) 

[sbordet]

Can you please take a JVM thread dump when that happens ?

BlockerTimeoutException is thrown when you have a blocking write that sits idle for a long period of time. This means that your client does not read the content that the server sends - at least not fast enough.

This is typical of stress/load tests: the server is so fast that the client cannot keep up.

Having said that, the server should have idle timed out before throwing BlockerTimeoutException.

If you can easily reproduce the issue, we are interested in trying to reproduce ourselves: can you share your code ?

[rocky-peng]

Thank you very much.
I tried to simulate this issue,but I failed and sometimes the exceptions like this

2016-10-28 15:28:25.212:WARN:oejs.HttpChannel:Scheduler-1336131694: Commit failed
java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
        at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:161)
        at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)        

I remember that the exceptions is about a deadlock sometimes,but I lost the log.

I also found that if the jetty start throwing those exceptions and more new requests come into jetty ,the jetty will be busy throwing those exceptions,so the new request can not be handled.

So is there any way to handle this more friendly and quickly ?

[sbordet]

If you are load testing and the server idle times out, you have a problem on the client that is not able to read what the server sends fast enough.

That Jetty throws those exceptions in sequence means that the client was overwhelmed by the traffic that itself generated, so it's only matter of writing a proper load test - and it's not as easy as it seems.

If you have a reproducible test case, please attach it here.

[laeubi]

This could be related to #517

[laeubi]

The main problem IMO is that when this exception is thrown it has happend somewhere else already on a different time.

SharedBlockingCallback is blocking until complete.await() finished and then check if the state is a exception to throw in this case it reaches the throw new IOException(_state); what includes the timeout exception set in the state...

[laeubi]

You might want to try out the fix here if it solves your case: #1037 if it only occurs under heavy load.

[joakime]

It's been over a year with no additional information. Closing.