Test Explorer fails to show tests. Tests output: "The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)"

[daiplusplus]

Description

I cloned a private Git repo on my local machine. The solution comprises of several projects all targeting .NET 4.6.2. I built the solution which pulled-in some NuGet packages, including MSTest.TestAdapter, 1.1.18 and MSTest.TestFramework, 1.1.18. There is a test project in the solution with several [TestClass]/[TestMethod] tests. The solution builds successfully, however the Test Explorer window is empty. On a coworker's machine running VS2015 the Test Explorer is populated correctly.

In the Output window's Test output I see the following:

------ Discover test started ------
The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)
========== Discover test finished: 0 found (0:00:00.1886778) ==========

Steps to reproduce

I have no specific reproduction steps - it happened spontaneously on my machine. Restarting VS, cleaning the project output and TestResults directory, and closing+reopening the Test Explorer window has no effect.

Expected behavior

The tests in the project should appear in Test Explorer.

Actual behavior

No tests appear in Test Explorer, and an error message in the Output window.

Environment

Please share additional details about the test environment.
Operating system, Build version of vstest.console, Package version of MSTest
framework and adapter

• Windows 10 x64 Enterprise, Creators Update
• Visual Studio 2017 Enterprise, 15.2
• MSTest package version 1.1.18
• Projects targeting .NET 4.6.2
• I have two copies of vstest.console installed:
  • C:\Program Files (x86)\Microsoft Visual Studio 15.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe - 15.0.26228.0 - 142KB
  • C:\Program Files (x86)\Microsoft Visual Studio 15.0\Common7\IDE\Extensions\TestPlatform\vstest.console.exe - 15.0.0.0 - 117KB

[AbhitejJohn]

@Jehoel: Sorry you are running into this. Few queries:

1. Are the test projects targeting x64 architecture? If yes, is Test -> Test Settings -> Default Process Architecture set to x64?
2. Can you set a system wide environment variable VS_UTE_Diagnostics to 1, restart VS and run through the scenario again please? This will output diagnostic logs in Output -> Tests pane in VS. Please do share that.

Edit by @smadala See following two comments.

#241 (comment)
#241 (comment)

[daiplusplus]

1. All projects are configured to build for AnyCPU. The "Prefer 32-bit" checkbox is greyed out.
   1.1. I changed "Default process architecture" to "x64" to see what would happen, and after building I see this in the Test Output window:

    ------ Discover test started ------
    Test run will use DLL(s) built for framework Framework45 and platform X64. Following DLL(s) will not be part of run: 
    MyAssembly.exe is built for Framework Framework45 and Platform X86.
     Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.
    The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)
    ========== Discover test finished: 0 found (0:00:00.2255871) ==========
   

   (MyAssembly.exe is a 4.6.2 AnyCPU project that does have "Prefer 32-bit" checked)
   (Interesting that it says Framework45 instead of Framework461, or is that irrelevant?)

2. I opened the VS2017 command-prompt and did this:

    **********************************************************************
    ** Visual Studio 2017 Developer Command Prompt v15.0.26430.16
    ** Copyright (c) 2017 Microsoft Corporation
    **********************************************************************
    C:\Program Files (x86)\Microsoft Visual Studio 15.0>SET VS_UTE_Diagnostics=1
    C:\Program Files (x86)\Microsoft Visual Studio 15.0>devenv
   

   When VS opened and I rebuilt the solution, there was no additional output in the Test Output window, only the same message as before:

    ------ Discover test started ------
    The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)
    ========== Discover test finished: 0 found (0:00:00.19589) ==========
   

[AbhitejJohn]

Thanks @Jehoel.

1. Yes, Framework45 is irrelevant.
2. You would need to set it for all applications, run

setx VS_UTE_Diagnostics 1

Looking at your issue again and the fact that this is working in VS2015, I suspect this could be because of a missing assembly(possibly in GAC in VS2015). Could you please turn on fusion logs and look for any assembly load failures? Here is how you could turn them on. You could zip the logs and share them over here as well.

[daiplusplus]

I now see more output in the Test window, it contains data I'd rather not publish publicly - can I message you on Skype-for-Business?

Here's the tail end of the output that seems the most relevant - no obvious errors though:

PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 
PowerShellTestContainerDiscoverer:IsTestFile - 

(etc - including more PowerShellTestContainerDiscoverer:IsTestFile - lines, some of which have valid filenames, others are empty)

PowerShellTestContainerDiscoverer:OnTestContainersChanged - FindPs1Files
test container discoverer executor://powershelltestexecutor/v1, discovered 0 containers
No containers found from 'PowerShellTools.TestAdapter.PowerShellTestContainerDiscoverer' :
test container discoverer executor://nodejstestexecutor/v1, discovered 0 containers
No containers found from 'Microsoft.NodejsTools.TestAdapter.TestContainerDiscoverer' :
test container discoverer executor://orderedtestadapter/v1, discovered 0 containers
No containers found from 'Microsoft.VisualStudio.MSTest.TestWindow.OrderedTestContainerDiscoverer' :
test container discoverer executor://generictestadapter/v1, discovered 0 containers
No containers found from 'Microsoft.VisualStudio.MSTest.TestWindow.GenericTestContainerDiscoverer' :
test container discoverer executor://webtestadapter/v1, discovered 0 containers
No containers found from 'Microsoft.VisualStudio.MSTest.TestWindow.WebTestContainerDiscoverer' :
DiscoveryOperation<DiscoverAllOrRunOnInitializeOperation> FinishedChangedCotainers, changed container count is 7
VirtualReadOnlyTestDataStore.OperationStateChanged State=ChangeDetectionFinished, operationInProgress=False
TestDiscoveryStats.OperationStateChanged State=ChangeDetectionFinished, InProgress=False
Discovering the following containers :
	C:\Git\MyProject\MyProject.CloudService\MyProject.FooWorkerRole\bin\Debug\MyProject.FooWorkerRole.dll
	C:\Git\MyProject\MyProject.Bar\bin\Debug\MyProject.Bar.exe
	C:\Git\MyProject\MyProject.Baz.Packager\bin\Debug\MyProject.Baz.Packager.dll
	C:\Git\MyProject\MyProject.Baz.Tests\bin\Debug\MyProject.Baz.Tests.dll
	C:\Git\MyProject\MyProject.Baz\bin\Debug\MyProject.Baz.dll
	C:\Git\MyProject\MyProject.SharedConfiguration\bin\Debug\MyProject.SharedConfiguration.dll
	C:\Git\MyProject\MyProject.StorageCli\bin\Debug\MyProject.StorageCli.exe
VirtualReadOnlyTestDataStore.OperationStateChanged State=DiscoveryStarting, operationInProgress=False
TestDiscoveryStats.OperationStateChanged State=DiscoveryStarting, InProgress=True
------ Discover test started ------
VirtualReadOnlyTestDataStore.OperationStateChanged State=DiscoveryStarted, operationInProgress=False
TestDiscoveryStats.OperationStateChanged State=DiscoveryStarted, InProgress=True
***    Discover TestCase using 'InMemoryUnitTestWriter' ***
RunSettings Content:
<RunSettings>
  <RunConfiguration>
    <ResultsDirectory>C:\Git\MyProject\TestResults</ResultsDirectory>
    <SolutionDirectory>C:\Git\MyProject\</SolutionDirectory>
    <TargetPlatform>X86</TargetPlatform>
    <TargetFrameworkVersion>Framework45</TargetFrameworkVersion>
  </RunConfiguration>
</RunSettings>
The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)
========== Discover test finished: 0 found (0:00:57.5407445) ==========
VirtualReadOnlyTestDataStore.OperationStateChanged State=OperationSetFinished, operationInProgress=False
TestDiscoveryStats.OperationStateChanged State=OperationSetFinished, InProgress=False
	Provider 'GroupByClassProvider' found 0 groups.
VirtualReadOnlyTestDataStore.OperationStateChanged State=DiscoveryFinished, operationInProgress=False
TestDiscoveryStats.OperationStateChanged State=DiscoveryFinished, InProgress=False
Pausing Queue..., current operation is: no current operation
About to Enqueue operation 'WaitForBuildOperation', hashcode:34282735 
Enqueue operation 'WaitForBuildOperation', hashcode:34282735 
Operation left in the the queue: 1
	'WaitForBuildOperation', hashcode:34282735


Processing Queue .....
Operation Dequeue : 'WaitForBuildOperation'
VirtualReadOnlyTestDataStore.OperationStateChanged State=OperationSetStarted, operationInProgress=False
TestDiscoveryStats.OperationStateChanged State=OperationSetStarted, InProgress=False
Firing BuildCompleted event...
Container Discoverer 'Microsoft.VisualStudio.TestWindow.VsAdapters.VsProjectOutputContainerDiscoverer' has container changes
VirtualReadOnlyTestDataStore.OperationStateChanged State=OperationSetFinished, operationInProgress=False
TestDiscoveryStats.OperationStateChanged State=OperationSetFinished, InProgress=False
test container discoverer executor://vsprojectoutputcontainerdiscoverer/v1, discovered 7 containers
Containers from 'Microsoft.VisualStudio.TestWindow.VsAdapters.VsProjectOutputContainerDiscoverer' :
	C:\Git\MyProject\MyProject.CloudService\MyProject.FooWorkerRole\bin\Debug\MyProject.FooWorkerRole.dll:executor://vsprojectoutputcontainerdiscoverer/v1
	C:\Git\MyProject\MyProject.Bar\bin\Debug\MyProject.Bar.exe:executor://vsprojectoutputcontainerdiscoverer/v1
	C:\Git\MyProject\MyProject.Baz.Packager\bin\Debug\MyProject.Baz.Packager.dll:executor://vsprojectoutputcontainerdiscoverer/v1
	C:\Git\MyProject\MyProject.Baz.Tests\bin\Debug\MyProject.Baz.Tests.dll:executor://vsprojectoutputcontainerdiscoverer/v1
	C:\Git\MyProject\MyProject.Baz\bin\Debug\MyProject.Baz.dll:executor://vsprojectoutputcontainerdiscoverer/v1
	C:\Git\MyProject\MyProject.SharedConfiguration\bin\Debug\MyProject.SharedConfiguration.dll:executor://vsprojectoutputcontainerdiscoverer/v1
	C:\Git\MyProject\MyProject.StorageCli\bin\Debug\MyProject.StorageCli.exe:executor://vsprojectoutputcontainerdiscoverer/v1
Container discoverer 'Microsoft.VisualStudio.TestWindow.VsAdapters.VsProjectOutputContainerDiscoverer' has no containers

[AbhitejJohn]

Thanks @Jehoel. As I suspected, these logs do not seem to be giving us much. Could you turn on fusion logging instead following the instructions above.

Looking at your issue again and the fact that this is working in VS2015, I suspect this could be because of a missing assembly(possibly in GAC in VS2015). Could you please turn on fusion logs and look for any assembly load failures? Here is how you could turn them on. You could zip the logs and share them over here as well

On

can I message you on Skype-for-Business?

Sure, you can send a mail to vstestsup@microsoft.com with any other details you might have.

[StingyJack]

@AbhitejJohn - what was the resolution for this?

Half of my tests are getting ignored in both VS and TFS.

[daiplusplus]

I generated the fusion logs - I had a quick look but I couldn't see any errors.

I do have another test project that works fine and its tests do appear in the Test Explorer window - so I compared the two projects. I noticed they're both .NET Framework 4.6.2 projects, however the working project has a reference to Microsoft.VisualStudio.QualityTools.UnitTestFramework which is being pulled from C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\PublicAssemblies\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll - whereas the non-working project has a reference to Microsoft.VisualStudio.TestPlatform.TestFramework which is being pulled from (solution root)\packages\MSTest.TestFramework.1.1.18\lib\net45\Microsoft.VisualStudio.TestPlatform.TestFramework.dll.

Indeed, the working project is using a standard Visual Studio Test project which came with its own VS-coupled references, while the non-working project is using the NuGet "MSTest V2" assemblies instead.

So why is MSTest V2 not working all of a sudden?

TL;DR Workaround:

1. Remove the NuGet package references and project assembly references to Microsoft.VisualStudio.TestPlatform.TestFramework.
2. Replace them with a reference to your Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll assembly instead, which is located inside your Visual Studio installation directory (Common7\IDE\PublicAssemblies).

[daiplusplus]

I've emailed vstestsup@microsoft.com with my fusion logs.

[daiplusplus]

My email to vstestsup@microsoft.com was rejected by the Microsoft e-mail server with this message:

Your message to VSADepT@microsoft.com couldn't be delivered.

The group VSADepT only accepts messages from people in its organization or on its allowed senders list, and your email address isn't on the list.

I see that VSADepT is not the same thing as VSTestSup - but I'm not sure if the intended recipients receive it or not - @AbhitejJohn did you get it?

[josteink]

TL;DR Workaround:

That may work locally, but won't fly on our CI server. A real proper resolution would be real nice :(

[josteink]

I found the reason for the failure on my end.

Basically some of my projects are really old, and have slowly been migrated to newer .NET-versions and test-frameworks.

Turns out that the app.config in my project I had the following section still having around, despite the project being "upgraded" to .NET 4.5.1 and latest MSTest:

  <system.codedom>
    <compilers>
      <compiler language="c#;cs;csharp" extension=".cs" type="Microsoft.CSharp.CSharpCodeProvider, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" warningLevel="4">
        <providerOption name="CompilerVersion" value="v3.5"/>
        <providerOption name="WarnAsError" value="false"/>
      </compiler>
    </compilers>
  </system.codedom>
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1" appliesTo="v2.0.50727">
      <dependentAssembly>
        <assemblyIdentity name="System.Web.Extensions" publicKeyToken="31bf3856ad364e35"/>
        <bindingRedirect oldVersion="1.0.0.0-1.1.0.0" newVersion="3.5.0.0"/>
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Web.Extensions.Design" publicKeyToken="31bf3856ad364e35"/>
        <bindingRedirect oldVersion="1.0.0.0-1.1.0.0" newVersion="3.5.0.0"/>
      </dependentAssembly>
    </assemblyBinding>
  </runtime>

Removing these made test-explorer able to discover and run my tests.

[AbhitejJohn]

@Jehoel: Yes, Replied.
@josteink: Thanks. <providerOption name="CompilerVersion" value="v3.5"/> would probably be the culprit.
@StingyJack: I suspect this would be a different issue. Could you please raise a new one with details?

[MartinF]

Just want to report that I had the same problems in a newly created project.
The app.config used to be empty, but after installing some packages assemblybinding's must have gotten added, which was causing the problem.

The solution was simple - I deleted the whole runtime section and cleared the output folders (bin, obj and TestResults), then it was working again.
Tried adding the assemblybindings again and got the same error.

[yesitspeter]

Just wanted to comment that I too am having the same issue with newly created projects. I'm on:

Microsoft Visual Studio Professional 2017
Version 15.3.4
VisualStudio.15.Release/15.3.4+26730.15
Microsoft .NET Framework
Version 4.7.02046

Update: I just took that test project, added the .Net standard library through NuGet and it worked. Strange!