Adapter only identifies tests in 3 of 5 of my exe files

[loki980]

We just started using google test a few weeks back. We'd been working with the command line output only. We found this extension and it seems to work correctly, but it only finds tests in 3 of the 5 exes I have implementing tests.

I started going through GoogleTestDiscoverer.cs to see if I can figure out what's wrong, but it takes the entire binary exe file as an argument? Can someone help me figure out how to start debugging this?

Thanks.

[loki980]

Each exe I have correlates to testing one class. We're using IBM Rational Rhapsody for development. The class tests that fail are classes that implement Rhapsody statecharts.

I stripped the files down to nothing and they still failed. The failure occurs when the class I'm testing inherits from OMReactive

I changed:
class shutdownMachine : public OMReactive, public iShutdownAck, public iShutdownRequest {
to:
class shutdownMachine : public iShutdownAck, public iShutdownRequest {

After this, my test is discovered just fine.

Any idea why this is occurring or how I can debug this?

[loki980]

OK, who knew I'd go down this rabbit hole. Only been using VS a week. Debugging Visual Studio within Visual Studio is not where I thought I'd end up.

I've been able to attach to the test discovery process and I see the array returned with and without the successfully scraped stdout. I'll update this after I continue debugging tomorrow.

Very nice readme by the way.

[csoltenborn]

Thanks for the kudos for the readme! Off the top of my head, I don't have any idea why your tests wouldn't be discovered... Have you seen the "How to debug" section of the readme? Attaching to the discovery process should be rather easy if using that MS extension.

Despite that, I guess we need some minimal example to help you out... If you cannot provide something like that, debugging is probably up to you :-) Here's what I would check: The tests are discovered by means of Dia, which allows to search executables for e.g. method signatures (as you probably have figured out already). We use a simple filter to get only the methods corresponding to actual tests (i.e., *::TestBody), and my first guess is that for some reason, that filter doesn't catch your tests. You could pass a different filter (i.e., *) into the DiaResolver.GetFunctions() method, put a breakpoint and investigate all methods found by Dia - that should help figure out why your tests are not found by the filter we use.

[csoltenborn]

Here's one more hint: If you manage to add your code to the SampleTests solution, it should be rather easy to add an according unit test (see GoogleTestDiscovererTests). This allows you to debug the adapter code without leaving your one instance of Visual Studio... And, btw, if this is a bug, and if you are willing to contribute and it's legally possibly to do so, we would happily add your test(s) to our test suite to make sure the bug is gone for good.

[loki980]

When executed from visual studio, the .dll my executable relies on (where the OMReactive class is found) is dynamically linked without issues (my environment in project properties, configuration properties, debugging is modified to append this location to PATH). When executed from ProcessLauncher::LaunchProcess, it cannot find this dll and the program fails saying so. I added the location of the dll to my path in windows and the extension finds everything as expected and can execute all my tests.

If you can figure out how to launch the process using the project's environment path, this would resolve this without having to manually register the dll or add it to our system path.

Thanks for your help and thanks for this extension!

[csoltenborn]

Thanks for debugging the issue and identifying the root cause! We will certainly look into this...

[csoltenborn]

I did some quick research: It probably is rather difficult to figure out the project's path, since discovery as well as execution run in different processes as VS itself, which means that (as far as I know) there is no clean way to access the VS instance.

However, there might be another approach: My guess is that if you enable "Copy local" in the properties of your references (which makes sure they end up in the same dir as the executable), they will be found by the ProcessLauncher. I'm not sure whether this is acceptable for you (although I think that this is a quite common project setup, but I'm far from being an expert in that area) - at least you don't need to change your test code or even the system path. Can you give this a try?

[loki980]

The dll isn't added as a reference. It's just added to the project's "configuration properties"->"linker"->"input"->"Additional Dependencies" line, so this won't work for me unless I change a bunch of things around. FYI - I'm on holiday for a week and this is the last thing I'm doing before turning off my PC, so I won't respond to anything for a while. Thanks again.

[csoltenborn]

@loki980 We have a solution for your problem (at least I hope so): We have added a new option "Path extension", the content of which will be appended to the PATH of the test executables. This way you should be able to add your additional dependencies to the test project without altering the system's PATH variable (and maybe even share these settings with your fellow developers).

You can download a VSIX containing the patch here - can you please give it a try and let us know if this works for you?

[loki980]

Thanks! I tested out the build - it does work. Adding the path allows discovering and running the tests. Removing causes the extension to basically do nothing. Perhaps you should catch the exception and give some feedback when the exe dies. It currently makes it look like your extension is broken, even though it's an issue with the exe.

[csoltenborn]

Which exception are you talking about? I wrote a unit test for the feature and tried to first make sure that tests are indeed not found when the DLL path is not configured as described, but was confronted with a windows dialog pointing out that there's a DLL missing (and it seems there is no way to prevent that dialog from showing up).

Is that the dialog you are talking about? If not, please let me know which exception I can catch and how that exception is propagated to the end user...

[loki980]

ProcessLauncher::LaunchProcess in the test discovery (and I assume the "run tests" part of the extension as well) throws a windows popup due to not finding the dll it needs. I assume you can catch this somehow? Your extension doesn't gracefully handle this and there's no explanation. I had to debug the extension to figure out why things weren't working.

[csoltenborn]

No, I apparently can't - just start your test executable from the console, and you'll get the same dialog. I will investigate this closer and see what I can do with the exit code, but the dialog's probably here to stay :-) :-|

[loki980]

You could identify when test discovery identifies an exe with no tests and display possible issues:

• Bad regex id'd a non-test executable
• No tests in the executable
• Executable failed to run correctly (Ask to confirm the exe runs from a command line. This is what failed for me. Running within VS was fine, but from a CL it died.)

Just trying to prevent someone (like I did) from spending a day debugging something where a message like the above can prevent it. I totally see this question popping up on my team at a later date. We have 3 positions open and I'm sure I'll get the question at least 3x - hah.

Thanks for your continued development on this extension. It's a livesaver for us. New gtest parsing that links directly to failing lines is sweet!

[csoltenborn]

That sounds like a good idea - would you mind to open an issue for that? I will then close this issue. Btw, the release will happen within a couple of days - I was basically waiting for feedback on #30, which I received from you now :-) (I know that you only talked about links to failed assertions, but that's good enough for me).