-
Notifications
You must be signed in to change notification settings - Fork 323
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
dotnet test is hanging after updating xUnit and test platform #10355
Comments
After a hint from @nohwnd I was able to figure out what is going on here. Given the message there was likely a tearing issue between DLLs here so I grabbed a binlog of the build of this test and noticed this: Parts of test platform are coming from 17.10 and others from 17.5. Changed around my references so that everything was 17.5 and the problem went away. This does raise the question though of why is this allowed? It seems there is an API dependency between these layers that is subject to tearing when the versions differ. Given that why don't we error or force exact NuGet matches here so that the tearing can't happen? I feel like the path that lead us here is one that customers are likely to follow. NuGet audit warnings lead us to upgrade our xunit references. Or incentive here is to upgrade to latest packages and that meant moving to xunit.runner.visualstudio 2.8.2. That in turn required us to move our Microsoft.TestPlatform.ObjectModel reference to 17.10.0. After that build succeeded and the majority of our tests ran. But our Microsoft.NET.Test.Sdk reference holds us back to 17.5.0 for testhost and now we have the tearing problem. |
Translation Layer is VisualStudio client this package is not normally referenced by test projects. In roslyn it is being used in LanguageServer and version is determined by The problem here is that the test.sdk is outdated (from Feb 2023), and the update brings In ideal case arcade should force you to update to newer or newest test.sdk, the same way package analyzer pushes you to newer xunit. Unfortunately we cannot force hard dependencies between test platform and xunit versions to force users to update. Once you update also test.sdk your tests will run:
In our new testing platform that xunit is onboarding to the version to use is defined by the dependency that xunit defines, and there is no additional "test.sdk" that needs to be kept in sync. So hopefully this situation will eventually get better. I agree that testhost is hard to debug, but first thing we would ask you to do is to run with diag logs:
Followed by connection loop and then
In the main (vstest.console) log. Meaning that we did not get through the initial protocol negotiation. |
By the look of the error, and seeing that only LanguageServer project failed, this is most likely caused by CommunicationUtilities library mismatch, rather than ObjectModel mismatch. Xunit does not depend on CommunicationUtilities, but TranslationLayer does. BUT translation layer is not commonly used by anyone except for VS, VSCode, Stryker and other IDE integration authors, so this is not a common scenario. |
Understood. But you can force hard dependencies between your own packages. Then if customers pick versions of xUnit and test platform that break this dependencies there will be a warning at restore time. |
The break happened because testhost and translation layer both ship communicationUtilities dll in the nuget, and translation layer overwrote the testhost dll with a newer version. So we would have to say: install only when other package is not installed at all or is installed with the same version. |
Why are they shipping private dependencies this way vs. making that a NuPkg they can both reference? |
They each normally live in a separate process, and we don't want others to take dependencies on the library that facilitates communication between the processes. Also because it has been done like that since forever. |
Description
dotnet test
is timing out after roslyn [pgraded it's xUnit references][roslyn-pr]Steps to reproduce
This is only happening in this project. Our other projects are executing just fine.
Expected behavior
The tests run.
Actual behavior
Test Run Aborted.
Diagnostic logs
After attaching to the testhost process I see the following exception:
With the stack trace
Errata
Getting to this stack trace was challenging and took hours of debugging and trying different ideas out. After putting
dotnet test
under a debugger it became clear thattesthost
is the process that was actually responsible for the problem. Buttesthost
cannot be run as an independent process because if you don't specify items like parentid it will crash on startup. That meant there was no reasonable way to debug it. If I starteddotnet test
and attached totesthost
when it started the exception had already happened. I spent significant time reading through the process options, generating diagnostics and none had any leads.Eventually I reached out in a internal channel and someone told me about setting
%VSTEST_HOST_DEBUG%=1
to enable child process debugging. I feel like that should not be a hidden environment variable but rather a first class option ondotnet test
. As someone who just went through this I really feel like there needs to be some breadcrumb in thedotnet test
arguments that show you a reasonable path to debugging thetesthost
process. Maybe even a first class switch like--debug-testhost
. Otherwise when it goes wrong there is no obvious way to debug it.The text was updated successfully, but these errors were encountered: