How to catch the threadpool RejectedException in the grpc server side

[dcrty90]

How to catch the threadpool RejectedException in the grpc server side? Reference issue: #636

For example, I have below setting for the grpc server executor: corePoolSize=2, maxPoolSize=4, queueCapacity=2. When the client has more than 6 requests, the server executor theadpool may throw TaskRejectedException, like:

[grpc-nio-worker-ELG-3-2-41] WARN NettyServerHandler:463 - Exception in onHeadersRead()
org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPollExecutor@6f636d4c[Running, pool size = 3, active threads = 3, queued tasks = 2, completed tasks = 0]] did not accept taks: io.grpc.internal.SerializingExecutor@6f636d4c

I have tried

1. catch the exception in my service class(extends XXXImplBase)

2. implement ServerInterceptor interface to override the onHalfClose()/onCancel method to catch exception

but both does't work.

Can someone please suggest where can I catch this kind of exception? Thanks a lot in advance.

[sanjaypujare]

Based on #636 it seems to be an open issue still. Also looking at NettyServerHandler.onHeadersRead() (your example) any exception like TaskRejectedException is wrapped inside an Http2Exception so you cannot catch it.

Unless @ejona86 has some ideas.... Also curious: what will you do after catching the exception?

[dcrty90]

Thanks @sanjaypujare for your response.

In this case, if the exception is not catched, the connection will be interruppted, then the left concurrent tasks will be cancelled. And no notifications will be sent to the client. In a word, some requests will be hang there.

If we have the ability to catch the exception, from the server side, we can response an error code. When clients receive the error code, it can pop up a retry window.

[sanjaypujare]

The wrapping Http2Exception would be sent to the client, right? Although it might be a generic error code can't that be made to work?

[ejona86]

Bad things happen if the executor throws, and there's no workaround other than "don't throw."

[ejona86]

We agree this is an issue, and I'm happy you are reporting that it is causing you trouble as it helps prioritize #636, but it seems just a duplicate of #636