std::cout goes to Test Explorer output window

[aey76]

Hello,
Consider this simple test:
TEST(aaa, bbb_2) {
EXPECT_EQ(10, 20) << "error 1";
for (int i = 0; i < 10 ; ++i) std::cout << "debug text line " << i << "\n";
EXPECT_EQ(10, 30) << "error 2";
}

The output of the test in the Test Explorer is:

The std::cout messages are not part of the xml output file.
Is it intended that the std::cout messages will be displayed in the Test Explorer output window?

[csoltenborn]

Yes - we think that if a test fails, all output it generated could be of interest. Do you object, or were you just worried about the difference between information provided by the xml output file and the test adapter?

Btw: You are going to see that output in the xml file produced by VsTestConsole.exe, too, so in a sense it contains more information than the xml file produced by Google Test itself...

[aey76]

I would suggest several things:

1. If it's important to capture all the print messages then GTA should also capture the prints that occurs before the first failure, right now this is not the case and for failure investigations it's more important to see the messages before the error.
2. I really think the output in the Test Explorer should present just the messages the user intent to print in case of errors. All the other cout's (before and after the error) may be irrelevant to the error and for that we have the console.
3. Other external tools that parse the XML output file (like jenkings) display just the error messages from the XML, that’s why all the important information about the error should be print from the ASSERT / EXPECT macros.

Maybe there are other considerations that I can't think of and if you still think this is the right behavior at least try to add configuration flag to capture all the messages or just the message from the ASSERT / EXPECT macros.

Thanks and regards.

[csoltenborn]

Ok, we have two issues here: First, we never noticed that the output happening before the first assertion is indeed not printed out - this is a bug which we will fix. Thanks for reporting!

Second, there are situations where the code to be tested prints out statements which are important for understanding the test's failure, and where this is out of control of the tester. That's why we still think that it's a good idea to print out the complete test output. We will however consider an option to switch this off.

[aey76]

OK, Thanks.
Switching this feature off will be very useful.
Thanks and regards.

[csoltenborn]

I was a bit too optimistic about being able to switch this off, I fear. Here's the thing:

• We parse a test executable's output as it runs. This allows us to report test results early, perform crashed test detection etc.
• Distinguishing a test's output from output produced by the code under test is not possible to our knowledge.
• The alternative would be to wait until the test executable has finished its execution and has written the test result xml file, and than parse that file. We certainly do not want to go that path as default behavior.

Thus, the only option appears to be offering that last behavior as an option. Before we do that, please elaborate on your use case - so far we haven't heard any complaints about our approach, so we would like to understand exactly why we would add that option.

[aey76]

Regardless of my specific use case, I thought the GTA purpose is to duplicate the MS Test behavior under the google test framework, if this is true then in MS Test all the outputs of Logger:: WriteMessage(…) goes out only to the Output window and not to the TestExplorer window so if we like to keep the same behavior we should do the same. More over I think people didn't complain about this because it didn't works correctly.

Although I think showing in the TestExplorer all the prints is wrong I trust yours opinion and yours decision.

Thanks and regards.

[csoltenborn]

Ok, thanks for your thoughts. The missing output bug is fixed with the upcoming 0.10.0 release. We will for now close this bug and see if the issue comes up again - an optional XML only mode seems possible (with some disadvantages as described).