Use RtlDllShutdownInProgress to detect process shutdown on Windows

[jkotas]

Switching to cooperative mode is not safe during process shutdown on Windows. Process shutdown can terminate a thread in the middle of the GC. The shutdown thread deadlocks if it tries to switch to cooperative mode and wait for the GC to finish in this situation.

Use RtlDllShutdownInProgress Windows API to detect process shutdown to skip cleanup that has to be done in cooperative mode.

The existing g_fProcessDetach flag is set too late - using this flag to skip cooperative mode switch would lead to shutdown deadlocks, and the existing g_fEEShutDown flag is set too early - using this flag to skip cooperative mode switch would lead to shutdown crashes.

Fixes #103624

[jkotas]

This is not 100% reliable fix, but it matches what we do in similar situations during shutdown currently.

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[jkotas]

Unfortunately, this fix does not work well. It replaces intermittent hang with intermittent crash during shutdown.

[jkoritzinsky]

What if we do the alloc context fixing during the destruction of the RuntimeThreadLocals object? Might require us to move the PreemptiveGCDisabled bool over first though.

[jkotas]

What if we do the alloc context fixing during the destruction of the RuntimeThreadLocals object?

I do not see how it would help.

This is the sequence that leads to the hang:

• OS initiates process shutdown
• There is a thread in the middle of the GC that gets killed by the process shutdown sequence. It means that the process state is indefinitely stuck in "GC in progress" state
• OS sends thread detach notification to the last thread in the process that runs the thread-local destructors. We try to switch to cooperative mode and wait for GC to finish - but that's never going to happen.

The problem with the current fix is that it is starts skipping the cleanup too early while there are still multiple threads running in the process. It means that we can end up running a GC without cleaned up alloc context that the GC gets very unhappy about.

I have been looking for a more precise detection of the moment after which it is fine to start skipping the cleanup.

[jkotas]

Ok, I think I found something that works. I have also fixed other instances of thread cleanup switching to cooperative mode that I found during testing and review.

[jkotas]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jkotas]

/azp run runtime-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[davidwrighton]

So, this is race condition city, and I think that there are still races that exist. Notably, I don't understand when the GC thread can be eliminated without running its cleanup logic, so I don't know how to analyze this for complete correctness, but based on the description you provided this should strictly reduce the set of deadlocks that are likely to occur.

[jkotas]

I don't understand when the GC thread can be eliminated without running its cleanup logic

Any shutdown with ongoing background activity is prone to this problem. What happens is:

• Some thread (typically the main app thread) decides to shutdown the process. It must switch to preemptive mode to do so.
• Some background thread starts GC
• Windows kernel suspends all threads (except the one thread running the shutdown) and kills them. The thread running in the middle of the GC can be one of those.
• The one remaining shutdown thread is resumed to proceed with the cleanup. If it ever tries to switch to cooperative mode and wait for GC to finish, it is going to hang.

I think that there are still races that exist.

Yes, I agree. The code has a lot of checks that should be either unnecessary or that have race conditions.

[Hrxn]

accurace -> accurate ?

[jkotas]

Thanks - fix included in #104318