core: rollback executor supplier, needs investigation

[YifeiZhuang]

No description provided.

[ejona86]

We saw a test failure with:

Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@123feec
java.lang.IllegalStateException: listener unset
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener.getListener(ServerImpl.java:796)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener.access$3000(ServerImpl.java:773)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:911)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializeReentrantCallsDirectExecutor.completeQueuedTasks(SerializeReentrantCallsDirectExecutor.java:67)
	at io.grpc.internal.SerializeReentrantCallsDirectExecutor.execute(SerializeReentrantCallsDirectExecutor.java:54)
	at io.grpc.internal.ServerImpl$ServerTransportListenerImpl.streamCreatedInternal(ServerImpl.java:641)
	at io.grpc.internal.ServerImpl$ServerTransportListenerImpl.streamCreated(ServerImpl.java:465)
	at io.grpc.inprocess.InProcessTransport$InProcessStream$InProcessClientStream.start(InProcessTransport.java:817)
	at io.grpc.internal.ForwardingClientStream.start(ForwardingClientStream.java:92)
	at io.grpc.internal.InternalSubchannel$CallTracingTransport$1.start(InternalSubchannel.java:681)
	at io.grpc.internal.ClientCallImpl.startInternal(ClientCallImpl.java:283)
	at io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:188)
	at io.grpc.census.CensusTracingModule$TracingClientInterceptor$1.start(CensusTracingModule.java:393)
	at io.grpc.census.CensusStatsModule$StatsClientInterceptor$1.start(CensusStatsModule.java:696)
	at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:332)
	at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:306)
	at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:218)

[YifeiZhuang]

io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@123feec
java.lang.IllegalStateException: listener unset

This log line might be caused by closing the stream but not set NoopListener yet.
SerializeReentrantCallsDirectExecutor running with InProcessTransport triggers this when everything runs inline, handleServerCall() not yet scheduled.

It might be causing TAP failures java.lang.AssertionError: Unexpected error logged. , but not sure others https://test.corp.google.com/ui#cl=381473612.

[ejona86]

Oh, directExecutor is causing work to be scheduled before the HandleServerCall Runnable is scheduled. Seems that requires triggering the UNIMPLEMENTED flow that calls stream.close(), since the application does not yet have access to the stream/call. And yeah, that could probably happen without directExecutor because of #7921 .

We'll need to re-evaluate our approach. We could just set the Listener in the UNIMPLEMENTED flow before cancelling the stream. We could instead add a SerializingExecutor.enqueue() that just calls runQueue.add(). We could do the same for SerializeReentrantCallsDirectExecutor, but it currently lazily initializes its queue. I have a suspicion that the taskQueue is generally allocated, so it may not hurt anything. And most alternatives seems to involve an allocation of their own.