Kestrel accept loop stops randomly and application stops processing new requests after migrating to dotnet 8/9 from dotnet 6.

[Piyush-022]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

After upgrading from .NET 6 to .NET 8/9, our ASP.NET Core application intermittently enters a “frozen” state in production where:

• The process remains alive (no crash, no restart)
• CPU usage drops to ~0%
• Memory usage remains stable
• The application stops accepting and processing new requests

A process dump taken during this state shows that Kestrel’s accept/dispatcher loop is no longer running (async accept loop state machines are missing), while the listening socket still exists and is marked as listening. as a result IIS started giving 502.3 as kestrel was not processing new proxied requests from IIS. The only way to recover application is restarting the app pool.

we are hosting application behind IIS using out-of-process model and ANCM.

This issue occurs rarely and randomly under productions random scenarios and load and has not been reproducible locally. First this issue happened in production env after 7-8 hours of deploying application with dotnet 8 over there. then we reverted application back to dotnet 6. and then we gave it second try and migrated to dotnet 9, but same issue happened after 15-20 days. this happened once again after 10 days of first occurrence.
nothing can be seen in event viewer as well. also the dump we have generated is with dotnet 9 only (from test environment). we don't have dump of process froze in production with dotnet 8.

Expected Behavior

Application should keep processing new requests as dotnet 6 because same application code was working perfectly with 6.

Steps To Reproduce

We are also not able to reproduce as this is very random.

Exceptions (if any)

Also got below kestrel related exception 4 times just before application stopped processing new requests. all these 4 exception thrown in span of milliSeconds.
{
"exception": {
"Message": "A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread.",
"HResult": -2146233088,
"ClassName": "System.AggregateException",
"InnerException": {
"StackTraceString": " at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportConnectionManager.RemoveConnection(Int64 id)\r\n at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection1.ExecuteAsync()", "Message": "No value found for the specified id.", "HResult": -2147024809, "ClassName": "System.ArgumentException", "RemoteStackIndex": 0, "Source": "Microsoft.AspNetCore.Server.Kestrel.Core", "ParamName": "id" }, "RemoteStackIndex": 0, "InnerExceptions": [ { "Message": "No value found for the specified id.", "RemoteStackIndex": 0, "Source": "Microsoft.AspNetCore.Server.Kestrel.Core", "StackTraceString": " at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportConnectionManager.RemoveConnection(Int64 id)\r\n at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection1.ExecuteAsync()",
"HResult": -2147024809,
"ClassName": "System.ArgumentException",
"ParamName": "id"
}
]
},
"data": {
"message": {
"data": "A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (No value found for the specified id. (Parameter 'id'))"
}
}
}

Also at the same time i got one occurrence of below exception as well. This is related to outgoing HTTP2 request

{
"exception": {
"Message": "A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread.",
"HResult": -2146233088,
"ClassName": "System.AggregateException",
"InnerException": {
"StackTraceString": " at System.Threading.Overlapped.GetOverlappedFromNative(NativeOverlapped* pNativeOverlapped)\r\n at System.Threading.Overlapped.Free(NativeOverlapped* nativeOverlappedPtr)\r\n at System.Net.Sockets.SocketAsyncEventArgs.Dispose()\r\n at System.Net.Sockets.Socket.Dispose(Boolean disposing)\r\n at System.Net.Sockets.NetworkStream.Dispose(Boolean disposing)\r\n at System.IO.Stream.Close()\r\n at System.Net.Security.AuthenticatedStream.Dispose(Boolean disposing)\r\n at System.Net.Security.SslStream.Dispose(Boolean disposing)\r\n at System.IO.Stream.Close()\r\n at System.Net.Http.Http2Connection.FinalTeardown()\r\n at System.Net.Http.Http2Connection.ReleaseStream()\r\n at System.Net.Http.Http2Connection.RemoveStream(Http2Stream http2Stream)\r\n at System.Net.Http.Http2Connection.Http2Stream.Complete()\r\n at System.Net.Http.Http2Connection.Http2Stream.Cancel()\r\n at System.Net.Http.Http2Connection.Http2Stream.OnReset(Exception resetException, Nullable1 resetStreamErrorCode, Boolean canRetry)\r\n at System.Net.Http.Http2Connection.Abort(Exception abortException)\r\n at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()", "Message": "Handle is not initialized.", "HResult": -2146233079, "ClassName": "System.InvalidOperationException", "RemoteStackIndex": 0, "Source": "System.Private.CoreLib" }, "RemoteStackIndex": 0, "InnerExceptions": [ { "StackTraceString": " at System.Threading.Overlapped.GetOverlappedFromNative(NativeOverlapped* pNativeOverlapped)\r\n at System.Threading.Overlapped.Free(NativeOverlapped* nativeOverlappedPtr)\r\n at System.Net.Sockets.SocketAsyncEventArgs.Dispose()\r\n at System.Net.Sockets.Socket.Dispose(Boolean disposing)\r\n at System.Net.Sockets.NetworkStream.Dispose(Boolean disposing)\r\n at System.IO.Stream.Close()\r\n at System.Net.Security.AuthenticatedStream.Dispose(Boolean disposing)\r\n at System.Net.Security.SslStream.Dispose(Boolean disposing)\r\n at System.IO.Stream.Close()\r\n at System.Net.Http.Http2Connection.FinalTeardown()\r\n at System.Net.Http.Http2Connection.ReleaseStream()\r\n at System.Net.Http.Http2Connection.RemoveStream(Http2Stream http2Stream)\r\n at System.Net.Http.Http2Connection.Http2Stream.Complete()\r\n at System.Net.Http.Http2Connection.Http2Stream.Cancel()\r\n at System.Net.Http.Http2Connection.Http2Stream.OnReset(Exception resetException, Nullable1 resetStreamErrorCode, Boolean canRetry)\r\n at System.Net.Http.Http2Connection.Abort(Exception abortException)\r\n at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()",
"Message": "Handle is not initialized.",
"HResult": -2146233079,
"ClassName": "System.InvalidOperationException",
"RemoteStackIndex": 0,
"Source": "System.Private.CoreLib"
}
]
}

What i understand is that non of the above exception should come because of out application code. i can be wrong here as well.

.NET Version

8.0.3 and 9.0.3

Anything else?

No response

[Piyush-022]

@davidfowl can you please review this issue.

[davidfowl]

we are hosting application behind IIS using out-of-process model and ANCM.

Why? Can you switch to in proc?

Strange that it's using HTTP/2 (though that might be red herring).

cc @halter73 @BrennanConroy

[BrennanConroy]

Kestrel’s accept/dispatcher loop is no longer running (async accept loop state machines are missing)

If that's true you should have a critical log with an exception: https://source.dot.net/#Microsoft.AspNetCore.Server.Kestrel.Core/Internal/ConnectionDispatcher.cs,74

[Piyush-022]

Kestrel’s accept/dispatcher loop is no longer running (async accept loop state machines are missing)

If that's true you should have a critical log with an exception: https://source.dot.net/#Microsoft.AspNetCore.Server.Kestrel.Core/Internal/ConnectionDispatcher.cs,74

@BrennanConroy @halter73 we were not having stdoutLogEnabled set to true in web.config at the time when this issue occurred. why i am saying that accept loop exited is because i didn't see it in dumpasync output in dotnet-dump analyze tool. also i captured running application's process dump twice, i can see accept loop task in both healthy dump.
also checked all stack using clrstack -all, only one kestrel related stack was there and this was for Heartbeat.TimerLoop. below is that stack

0000003BC6F3F758 00007ff85c4b0b24 [InlinedCallFrame: 0000003bc6f3f758] System.Threading.Monitor.<Wait>g____PInvoke|12_0(System.Runtime.CompilerServices.ObjectHandleOnStack, Int32)
0000003BC6F3F758 00007ff7e9ddda9b [InlinedCallFrame: 0000003bc6f3f758] System.Threading.Monitor.<Wait>g____PInvoke|12_0(System.Runtime.CompilerServices.ObjectHandleOnStack, Int32)
0000003BC6F3F700 00007FF7E9DDDA9B System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ManualResetEventSlim.cs @ 578]
0000003BC6F3F840 00007FF7EA321121 System.Threading.ManualResetEventSlim.Wait(System.TimeSpan) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ManualResetEventSlim.cs @ 404]
0000003BC6F3F870 00007FF8417943F7 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat.TimerLoop() [/_/src/Servers/Kestrel/Core/src/Internal/Infrastructure/Heartbeat.cs @ 74]
0000003BC6F3FAD8 00007ff846292d63 [DebuggerU2MCatchHandlerFrame: 0000003bc6f3fad8]

also dumpheap don't show any object of ConnectionDispatcher, whereas in healthy application it's there.

if you want me to check something in dump, please let me know.

[Piyush-022]

@BrennanConroy @halter73 can anyone please have a look at above comment?

also @davidfowl we are forcing client to use http version 2.0 using HttpRequestMessage.Version

[davidfowl]

also @davidfowl we are forcing client to use http version 2.0 using HttpRequestMessage.Version

That second stack is an outgoing http request then, nothing to do with kestrel.

[JuergenAuer]

@Piyush-022

(1) Add a working log, may be a simple StreamWriter, that writes informations into a file with timestamp. Good: One file per request.

(2) Add a FirstChanceException Handler, that writes every exception into an own file.

Comparing the timestamps it should be possible to localize the error.

And you have the exact error.

[Piyush-022]

That second stack is an outgoing http request then, nothing to do with kestrel.

@davidfowl Are there any other possible reasons that could cause this issue? We are not modifying or doing anything that would directly interfere with Kestrel’s internal connection management.

The reason I shared the HTTP-related exception is because it also involves sockets/connections and we got this at same time with kestrel exceptions. In this case, we see an exception during connection disposal indicating that the handle is not initialized or may have already been freed.

Similarly, on the inbound side, we are seeing exceptions from Kestrel when removing requests where the specified connection ID does not exist or may be was already removed. Because both issues involve connection/socket lifecycle management, I was wondering if they could be related.

[Piyush-022]

@BrennanConroy @halter73
Does anyone have any idea why this issue occurs only in .NET 8/9 and not in .NET 6? I’ve shared my reasoning on why I think the accept loop stops in this comment: #64931 (comment)

[BrennanConroy]

No, enable logging so we don't have to guess randomly.