Run All fails with NotImplementedException

[JesseOlmer]

Using the Run All button in Test Explorer fails to execute tests with the following stack in Test Output. Highlighting all tests in Test Explorer, right-clicking and using "Run Selected Tests" successfully executes all tests.

Visual Studio 2015 Community Edition, Update 2.

ERROR: Exception while running tests: System.NotImplementedException: The method or operation is not implemented.
   at Dia.IDiaSymbol.findChildren(SymTagEnum symTag, String name, UInt32 compareFlags, IDiaEnumSymbols& ppResult)
   at GoogleTestAdapter.DiaResolver.DiaResolver.FindFunctionsByRegex(String pattern)
   at GoogleTestAdapter.DiaResolver.DiaResolver.GetFunctions(String symbolFilterString)
   at GoogleTestAdapter.TestCases.TestCaseResolver.FindTestCaseLocationsInBinary(String binary, List`1 testMethodSignatures, String symbolFilterString, String pathExtension, List`1 errorMessages)
   at GoogleTestAdapter.TestCases.TestCaseResolver.ResolveAllTestCases(String executable, List`1 testMethodSignatures, String symbolFilterString, String pathExtension, List`1 errorMessages)
   at GoogleTestAdapter.TestCases.TestCaseFactory.GetTestCaseLocations(IList`1 testCaseDescriptors, String pathExtension)
   at GoogleTestAdapter.TestCases.TestCaseFactory.CreateTestCases()
   at GoogleTestAdapter.GoogleTestDiscoverer.GetTestsFromExecutable(String executable)
   at GoogleTestAdapter.TestAdapter.TestExecutor.GetAllTestCasesInExecutables(IEnumerable`1 executables)
   at GoogleTestAdapter.TestAdapter.TestExecutor.RunTests(IEnumerable`1 executables, IRunContext runContext, IFrameworkHandle frameworkHandle)

[csoltenborn]

I've just updated my VS 2015 Community Edition to Update 2, and I cannot reproduce the issue. What's weird is that the code to resolve test case locations is the same, no matter whether all or a selection of tests is executed...

Can you please let us know the version of your msdia140.dll (to be found at C:\Program Files (x86)\Microsoft Visual Studio 14.0\DIA SDK\bin)?

As a workaround, you can disable symbol information parsing at the Google Test Adapter General options (resulting in no source locations of your tests, but you will still get links to failed assertions etc.).

[JesseOlmer]

msdia140.dll is 14.0.23918.0

For full disclosure, the other issues I'm hitting with GoogleTestAdapter (and not the built-in test adapters) are:

1. My test assembly pdb files remain locked after a test run (by TE.ProcessHost.Managed.exe)
2. After building I have the following error:

------ Discover test started ------
Test run will use DLL(s) built for framework Framework45 and platform X64. Following DLL(s) will not be part of run: 
test.exe is built for Framework None and Platform X86.
 Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.
GoogleTest: Does D:\path\to\test.exe match [Tt]est[s]{0,1}.*.exe: true
Found 7 tests, resolving symbols
Loading symbols from D:\path\to\test.exe
From D:\path\to\test.exe, found 0 symbols in 42 ms
Couldn't find 7 symbols in D:\path\to\test.exe, looking from DllImports in module directory D:\path\to
D:\path\to\test.exe matches regex '[Tt]est[s]?\.exe'
ERROR: Exception while discovering tests: System.NotImplementedException: The method or operation is not implemented.
   at Dia.IDiaSymbol.findChildren(SymTagEnum symTag, String name, UInt32 compareFlags, IDiaEnumSymbols& ppResult)
   at GoogleTestAdapter.DiaResolver.DiaResolver.FindFunctionsByRegex(String pattern)
   at GoogleTestAdapter.DiaResolver.DiaResolver.GetFunctions(String symbolFilterString)
   at GoogleTestAdapter.TestCases.TestCaseResolver.FindTestCaseLocationsInBinary(String binary, List`1 testMethodSignatures, String symbolFilterString, String pathExtension, List`1 errorMessages)
   at GoogleTestAdapter.TestCases.TestCaseResolver.ResolveAllTestCases(String executable, List`1 testMethodSignatures, String symbolFilterString, String pathExtension, List`1 errorMessages)
   at GoogleTestAdapter.TestCases.TestCaseFactory.GetTestCaseLocations(IList`1 testCaseDescriptors, String pathExtension)
   at GoogleTestAdapter.TestCases.TestCaseFactory.CreateTestCases()
   at GoogleTestAdapter.GoogleTestDiscoverer.GetTestsFromExecutable(String executable)
   at GoogleTestAdapter.GoogleTestDiscoverer.DiscoverTests(IEnumerable`1 executables, ITestFrameworkReporter reporter)
   at GoogleTestAdapter.TestAdapter.TestDiscoverer.DiscoverTests(IEnumerable`1 executables, IDiscoveryContext discoveryContext, IMessageLogger logger, ITestCaseDiscoverySink discoverySink)
========== Discover test finished: 7 found (0:00:01.9906745) ==========

[csoltenborn]

Ok, the Dia Dll is the same as on my machine.

We haven't seen any issue like this afaik, and some of the log output is not generated by the test adapter itself. Any chance that you can share the solution containing your 7 tests?

[csoltenborn]

For the record: replacing some of the implemented method bodies in DiaMemoryStream with throw new NotImplementedException(); results in different exceptions (x86 as well as x64):

ERROR: Exception while discovering tests: System.Runtime.InteropServices.COMException (0x806D0004): Ausnahme von HRESULT: 0x806D0004
bei Dia.IDiaDataSource.loadDataFromIStream(IStream pIStream)...

[jgefele]

Hi @JesseOlmer,

the debug output ("GoogleTest: Does D:\path\to\test.exe match", "Found 7 tests, resolving symbols") doesn't look like our debug output, but rather those of the test runner we forked / ported to C#: github, VS gallery

Are you sure you're reporting to the correct project?

Cheers,
Jonas

[JesseOlmer]

Hi @jgefele,

I'm running "Google Test Adapter" from the VS Extension Gallery. You and Christian are listed as the authors, version is 0.5.1.345.
I'm using Google Test from https://github.com/google/googletest which I compiled locally on the machine experiencing this issue.

@csoltenborn I'll put together a stripped-down project tonight that will hopefully repro the issue.

Also, a slight correction to an earlier comment of mine: My test PDB files are locked after test discovery after a build, not only after an actual test run.

Thanks!

[csoltenborn]

@JesseOlmer Please make sure that you do not have both test adapters installed for some reason! If you search for resolving within our repository here at GitHub, you don't get any hit; if you do the same at the original repository, you get this hit:

logger.SendMessage(TestMessageLevel.Informational, sprintf "Found %d tests, resolving symbols" (testCases.Length))

The according line in our code looks like this:

TestEnvironment.LogInfo("Found " + testCases.Count + " tests in executable " + executable);

Same for the search term loading. That makes me think that Jonas might be right...

[jgefele]

In this commit the logging output has changed and the lines mentioned above have been removed.

[jgefele]

I checked again. These lines of your output seem to be from GoogleTestRunner (the other GoogleTest extension)

GoogleTest: Does D:\path\to\test.exe match [Tt]est[s]{0,1}.*.exe: true
Found 7 tests, resolving symbols
Loading symbols from D:\path\to\test.exe
From D:\path\to\test.exe, found 0 symbols in 42 ms
Couldn't find 7 symbols in D:\path\to\test.exe, looking from DllImports in module directory D:\path\to

However these lines look like our extension:

D:\path\to\test.exe matches regex '[Tt]est[s]?\.exe'
ERROR: Exception while discovering tests: System.NotImplementedException: The method or operation is not implemented.
   at Dia.IDiaSymbol.findChildren(SymTagEnum symTag, String name, UInt32 compareFlags, IDiaEnumSymbols& ppResult)
   at GoogleTestAdapter.DiaResolver.DiaResolver.FindFunctionsByRegex(String pattern)
   at GoogleTestAdapter.DiaResolver.DiaResolver.GetFunctions(String symbolFilterString)
   at GoogleTestAdapter.TestCases.TestCaseResolver.FindTestCaseLocationsInBinary(String binary, List`1 testMethodSignatures, String symbolFilterString, String pathExtension, List`1 errorMessages)
   at GoogleTestAdapter.TestCases.TestCaseResolver.ResolveAllTestCases(String executable, List`1 testMethodSignatures, String symbolFilterString, String pathExtension, List`1 errorMessages)
   at GoogleTestAdapter.TestCases.TestCaseFactory.GetTestCaseLocations(IList`1 testCaseDescriptors, String pathExtension)
   at GoogleTestAdapter.TestCases.TestCaseFactory.CreateTestCases()
   at GoogleTestAdapter.GoogleTestDiscoverer.GetTestsFromExecutable(String executable)
   at GoogleTestAdapter.GoogleTestDiscoverer.DiscoverTests(IEnumerable`1 executables, ITestFrameworkReporter reporter)
   at GoogleTestAdapter.TestAdapter.TestDiscoverer.DiscoverTests(IEnumerable`1 executables, IDiscoveryContext discoveryContext, IMessageLogger logger, ITestCaseDiscoverySink discoverySink)

@JesseOlmer: Could you please double check that you have no other GoogleTest related extension installed/enabled and retry in order to isolate the issue?

@csoltenborn: When we get the stripped-down project, can you try to reproduce it? I still haven't my Windows machine running again...

[JesseOlmer]

My apologies, I did have both extensions installed.
I've tried a few different things with this in mind and have experienced the following:
With GoogleTestRunner enabled, and GoogleTestAdapter disabled, everything works fine (test discovery, not locking the PDB, Run All).
With GoogleTestAdapter enabled, and GoogleTestRunner disabled, everything remains broken.

I'll work on that stripped-down project in a few hours and get that out to you, unless you think somehow GoogleTestRunner is breaking GoogleTestAdapter even when disabled?

[csoltenborn]

@jgefele Sure, I will give it a try... If GoogleTestRunner indeed breaks our adapter when installed in parallel, we have at least learned something. @JesseOlmer Thanks for your efforts!

[JesseOlmer]

Stripped down project is at https://github.com/JesseOlmer/gta-demo. Just needs your own gtestd.lib and includes.

I went ahead and uninstalled both extensions, restarted, re-installed GTA, restarted again, and am still seeing the issues.

[csoltenborn]

@JesseOlmer Thanks for the sample solution! I can indeed reproduce the problem now... stay tuned.

[csoltenborn]

Here's a quick update of what we think we have understood so far:

• It seems that our code is not (directly) responsible for the exception
• An earlier Dia library does not throw an exception, but doesn't find your test either

Thus, we think that there's something special about the PDB generated by your solution. If you have any idea on that, please let us know - we haven't seen anything obvious with your solution and project settings.

We will continue investigation, but that'll take some time. In the meantime, we will release an intermediate version which will ignore the exception and behave as if the Parse symbol information option is set to false. Additionally, we will make sure that the PDB is never locked by our code (that's indeed what one could consider a bug).

[JesseOlmer]

The only "specialization" from the default VS2015 "empty project" settings are

• Change the runtime library setting from /MDd to /MTd
• Link to the gta-demo project (also /MTd), gtestd.lib & reference the googletest includes

Please keep in mind in case it helps diagnosis that everything functions perfectly with GoogleTestRunner against the same project. I'm not familiar enough with either to understand if that's simply them swallowing exceptions more, but hopefully it'll help you guys out.

I'm happy to help further diagnosis and any potential fixes, but at this point I think I'll have to leave your extension disabled in favor of the Runner extension. Best of luck!

[csoltenborn]

Just to make sure that we're talking about the same test runner: I guess you have installed the Google Test Runner for Visual Studio 2015 by Valentin Kantchev, V1.8.1? And that adapter provides source locations (i.e., it finds the single test of your solution and - if the test is selected - displays source file and line number as a link (and moves you to the test's location if clicked))?

I'm asking because I just installed that adapter: Ony my machine, it discovers your test, but also does not provide the source location. In my case, it just displays a link named 'Testbody', which has no effect when clicked. Thus, it seems to have the exaxt same issues as our adapter, but seems to swallow the exception (and doesn't lock the pdb). Here's a screen shot of that adapter on my machine:

And a screenshot of our adapter (my local, already patched version, but you should get the same behavior if switching off Parse symbol information):

Note that I'm not trying to talk you into our adapter (we do not have any commercial interests, as you of course know) - I'm just trying to figure out whether analyzing that adapter's implementation (which we anyways know pretty well, since it was the starting point for our adapter) is worth the effort.

[csoltenborn]

For the record: Problem does not seem to be related to VS Update 2 - I can also produce it on VS Update 1.

[JesseOlmer]

@csoltenborn Yes, I have the bad "TestBody" link with GTR. I didn't mean to offend or criticise your work, it was a poorly worded attempt to let you know that I'm happy to try any fixes you have with the caveat that GTR would remain installed on my system in parallel since it met my needs.

[csoltenborn]

@JesseOlmer Don't worry, I didn't understand it that way... especially after you went the extra mile to provide a really minimal example solution! I just wanted to figure out if the other adapter indeed works on your system as desired. We'll let you know as soon as we have something to test :-)

[csoltenborn]

I think I have identified the problem. Since we figured that something is weird about the pdb generated by your example solution, I installed the Debugging Tools for Windows and used dbh to take a closer look. Executing the command dbh test.pdb dump >test.pdb.txt resulted in a file with the following content:

error 0x0 calling SymEnumSymbols()

Perfoming the same on our sample solution's Test.pdb results in a file >1MB. I then compared your project's settings with ours and identified Linker/Debugging/Generate debug info as the cause. Our setting is true, your setting is Optimize for faster linking (/DEBUG:FASTLINK). Changing the latter to true solves the problem, and the dump generated by dbh is now >600k.

If nobody has any objections, I will close this issue and add according information to our docs (although future cases will only suffer from missing source locations and traits - no more locked pdbs or undiscovered tests)...

[jgefele]

Nice! (well... that you found the issue)

Here is the related bug report for DIA140.

[csoltenborn]

@JesseOlmer If you want to try something, feel free to download this version of our adapter. It should - even with your project settings - find the tests, but still doesn't provide source locations (since there are none in the pdb).