Extract stack traces of hanging CI test runs

[weltkante]

Is your feature request related to a problem? Please describe.
Hangs in CI build are hard to diagnose, especially if they are not reproducable locally or do not happen deterministically.

Describe the solution you'd like and alternatives you've considered
dotnet-dump allows collecting dumps of a process and analyze them

PR #3056 started taking screenshots of hanging CI test runs, I think it would be even better if we used dotnet-dump to take dumps

Will this feature affect UI controls?
No, test infrastructure only

[weltkante]

Assuming the testing process id can be figured out, following powershell script takes a minidump (so storage size should be no concern) and extracts the stack traces. I also include the thread summary because it mentions apartments and lock counts.

dotnet-dump collect --process-id $TargetProcessId --type mini --output hangdump
"clrthreads`nclrstack -all`nexit`n" |& dotnet-dump analyze hangdump > hangdump.txt

The CI build may also need to install dotnet-dump if arcade doesn't do that yet, see linked doc page.

[RussKie]

I've been mulling over how we can get a memory dump. Your proposal is one of several ways, assuming dotnet-dump is installed, or it can be installed as part of a build run.

We'll need to look at this in conjunction with dotnet/arcade#5398.

[RussKie]

Closing as we can get crash dumps.

[weltkante]

@RussKie do we actually get dumps from hangs? The OS will only create them for crashes. I guess its no priority anymore since there are no unidentified hangs remaining and we hopefully won't add new ones ;-)

[RussKie]

We can revisit it, if necessary. Keeping my 🤞 that we won't need this.

[RussKie]

FYI I'm working on this, I have a prototype (4310c3c) that works locally, trying to sort out the CI leg