Options have no effect on run selected?

[Eistroll]

Hi there
I have disabled break on failure and enabled catch exceptions in the options but when I invoke "Run selected test" the options are not used.
Executing command 'base_test.exe --gtest_output="xml:C:\Users\thomas.roller\AppData\Local\Temp\tmp7548.tmp" --gtest_catch_exceptions=0 --gtest_break_on_failure=1 --gtest_filter=TestCOPManagerIterator.BackwardBasics'

Am I doing something wrong?

[csoltenborn]

Nothing that I can think of. I didn't have time to try to reproduce this yet - can you please try to configure your settings, restart VS, and then try again without changing the settings after restart? Maybe it's a refresh issue... And here's some more questions: Do you have Test/Test Settings/Keep test execution engine running enabled? Is this also happening if you change the options through the adapter's toolbar? And finally: Does it always work if you Run all tests?

[Eistroll]

• Test/Test Settings/Keep test execution engine running enabled? Is on but has no influence if I turn it off.
• No difference if I use toolbar.
• If I use the filter of the test explorer and make Run all then it also does not work

When I restart VS then it uses the options so it seems to be a refresh issue

[csoltenborn]

We have just finished our first debug session on this issue, and it looks like the VS test execution engine only asks our run settings service for fresh settings if "Run All" (without any filter) is clicked. This is just weird - we are pretty sure that in the past, the test execution engine would ask for fresh settings at every test run, no matter whether a selection of tests or all tests were executed.

Currently, this looks like a regression in VS. We see two "workarounds":

• Always use "Run all tests"
• Restart VS after changing options of our test adapter

Both workarounds obviously suck big time :-(

We will now try to figure out the VS version at which that behavior has changed, and also if it is an VS API change (the test adapter API is not official, so we have to figure all that stuff out on our own) - obviously, MS has worked on the test framework (see the new "Run tests in parallel" button), so maybe this is new behavior to which we have to adapt.

If you want to help, and if you happen to have access to VS 2015 without updates or VS 2015 with just Update 1, please let us know whether our options still work with one of these versions.

[Eistroll]

I am sorry. We only have VS2015 with Update 2 around here...

[csoltenborn]

I had the chance to try different VS versions today. Here's the thing: It works just fine on VS Enterprise Edition 2015 without update, with update 1, and with update 2.

Thus, it seems that the problem only occurs with the Community Edition and (possibly) the Professional Edition.

@Eistroll Can you confirm that you are not using the Enterprise Edition?

If this is indeed the case, the only thing we can probably do about this issue is filing a bug report against the affected VS editions - it currently doesn't look like it's our fault...

On the plus side, I think that I have found a slightly better workaround: Close and re-open your solution. This apparently also triggers reloading of the settings. Still not nice at all, but better than the other workarounds we suggested.

[csoltenborn]

@Eistroll Which VS edition are you guys using? I assume it's not the Enterprise edition?

[Eistroll]

Hi Christian,
we are using VS 2015 Professional and no Enterprise edition

[csoltenborn]

I have reported the problem to MS, here's the bug report. Feel free to add more information at that place as needed... (and maybe vote for the bug).

[csoltenborn]

Looks like our assumption was right - the bug has been closed with the comment that this will be fixed with update 3. Thanks, @Eistroll, for reporting this!

[Eistroll]

Thanks for the info and investigating it.

[csoltenborn]

@Eistroll The release candidate for VS update 3 has been released, and seems to have fixed the issue (at least on Community edition) - depending on how badly the regression hurts you guys, you may want to install it already.

[Eistroll]

Hi there,

thanks for the information, but I think we will wait :-)

On Thu, Jun 16, 2016 at 6:30 PM, Christian Soltenborn <
notifications@github.com> wrote:

@Eistroll https://github.com/Eistroll The release candidate for VS
update 3 has been released
https://www.visualstudio.com/en-us/news/releasenotes/vs2015-update3-vs,
and seems to have fixed the issue (at least on Community edition) -
depending on how badly the regression hurts you guys, you may want to
install it already.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#50 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AQXgYiUloHcmi9WsastfySHPMiQJQjXtks5qMXoZgaJpZM4IVKMr
.