Getting AppDomainUnloadedExceptions since the update to 17.11

[cremor]

Description

I've recently updated my build machine from Visual Studio 17.10.6 to 17.11.2 and the Azure DevOps build agent to 3.243.1. Now I'm getting multiple AppDomainUnloadedExceptions in the VSTest build step logs of my integration tests. These exceptions do not fail tests or the build, but are still reported as errors on the Azure DevOps build result page.

The test build step even contains the log line ##[error]Test Run Failed., but the step is not actually marked as failed. (Which is good, because no tests failed.)

It seems like I only get those exceptions when the tests are run on the build agent. I haven't reproduced them on my local machine.

I also only get them when I run (Oracle database) integration tests. Unit tests do not have any problems.
Note: An Oracle method shows up in the exception stack trace. But I haven't updated the Oracle NuGet package in a while. So the exceptions only started after updating VSTest.

Does VSTest 17.11 maybe unload some AppDomains during execution, which up to 17.10 didn't happen?

This is the exception:

##[error]Unhandled Exception: System.AppDomainUnloadedException: Attempted to access an unloaded AppDomain.
##[error]   at System.Threading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)
##[error]   at System.Threading.Thread.Start()
##[error]   at OracleInternal.ConnectionPool.PoolManager`3.DedicatedThreadFunc(Object state)
##[error]   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
##[error]   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
##[error]   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
##[error]   at System.Threading.ThreadHelper.ThreadStart(Object obj)

Steps to reproduce

Sorry, but I'm not sure what exactly triggers this. So those steps are an assumption.

1. Have a self-hosted build agent with Visual Studio 17.11.2 and Azure DevOps build agent 3.243.1.
2. Have an Azure DevOps build pipeline with a VSTest@2 build step.
3. Have a test project with multiple Oracle integration tests using the Oracle.ManagedDataAccess NuGet package.
4. Run the tests on the build agent.
5. Check the build logs and/or build result page.

Expected behavior

No exceptions.
Or, if the exceptions actually happen but don't matter: No build errors on the build result page.

Actual behavior

Exceptions are logged and shown as build errors on the Azure DevOps build result page.

Diagnostic logs

Please tell which log lines you need. It's very hard to anonymize the whole log file.
Here are a few lines which I think might be important:

Running tests using vstest.console.exe runner.
======================================================
[...]
Run the tests locally using vstest.console.exe
========================================================
[...]
##[debug]Found file version info for vstest.console.exe present in C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\IDE\Extensions\TestPlatform to be 17.11.0-release-24352-05
##[debug]Forcing TPv2 as the vstest.console.exe pointed to is the TPv2 one.
##[debug]Set DTA.ForcePlatformV2 to True.
##[debug]Set DTA.TestPlatformVersion to 17.11.0-release-24352-05.
[...]
**************** Starting test execution *********************
##[debug]Generating list of args to pass to vstest.console.exe.
C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\IDE\Extensions\TestPlatform\vstest.console.exe "@D:\agent\_work\_temp\p0rlvzhibgs.tmp"
VSTest version 17.11.0 (x64)
[REMOVED: list of all assemblies containing "test" in their name]
/TestCaseFilter:"TestCategory!=External&TestCategory!=Office"
/Settings:"D:\agent\_work\_temp\xjavzyr1dsh.tmp.runsettings"
/Logger:"trx"
/TestAdapterPath:"D:\agent\_work\3\s"
/Diag:"D:\agent\_work\_temp\Attempt--1_a1lfr5.diag"
Starting test execution, please wait...
Logging Vstest Diagnostics in file: D:\agent\_work\_temp\Attempt--1_a1lfr5.diag
A total of 14 test files matched the specified pattern.
Blame: Attaching crash dump utility to process testhost.net48 (6068).
Data collection : Logging DataCollector Diagnostics in file: D:\agent\_work\_temp\Attempt--1_a1lfr5.datacollector.24-09-03_09-44-37_38229_4.diag
Logging TestHost Diagnostics in file: D:\agent\_work\_temp\Attempt--1_a1lfr5.host.24-09-03_09-44-38_28863_4.diag
[...]
Blame: Attaching crash dump utility to process testhost (4136).
Data collection : Logging DataCollector Diagnostics in file: D:\agent\_work\_temp\Attempt--1_a1lfr5.datacollector.24-09-03_09-51-30_95235_11.diag
Logging TestHost Diagnostics in file: D:\agent\_work\_temp\Attempt--1_a1lfr5.host.24-09-03_09-51-31_79611_11.diag
No test matches the given testcase filter `TestCategory!=External&TestCategory!=Office` in D:\agent\_work\3\s\REMOVED.dll
Results File: D:\agent\_work\_temp\TestResults\dotnet-build$_dotnet-build_2024-09-03_09_44_57.trx
##[error]Test Run Failed.
##[debug]Processed: ##vso[task.logissue type=error;]Test Run Failed.
Total tests: 3815
     Passed: 3811
    Skipped: 4
 Total time: 6.9435 Minutes
##[debug]Exited vstest.console.exe with code 0.
##[debug]PERF: ExecuteVsTestPhase.InvokeVSTest: took 418617.4686 ms
Vstest.console.exe exited with code 0.
**************** Completed test execution *********************

Environment

Windows Server 2019
Visual Studio 2022 17.11.2
Azure DevOps build agent 3.243.1
VSTest version 17.11.0 (x64)
VSTest@2 build step
.NET Framework 4.8 runtime

[nohwnd]

I think you are hitting this error: #5184

The stack trace does not look like it is produced by us, and so you maybe had it reported to errors before, but it was silently ignored. In the linked issue all information reported to error stream will fail the run.

[cremor]

Thanks. I can confirm that setting VSTEST_DISABLE_STANDARD_OUTPUT_FORWARDING=1 resolves the issue for me.
I'll close this issue and watch #5184 for updates.