Google Test fails in GTA but passes on the command line.

[KevinLucidyne]

Occurs on latest version 0.7.0.452 as well as previous versions.

To reproduce the error:

Download and compile this code on a virgin copy of VS 2013:
https://github.com/BVLC/caffe/tree/windows

Download and run the VSIX Installer from
https://github.com/csoltenborn/GoogleTestAdapter/releases/download/v0.6.0/GoogleTestAdapter-0.6.0.vsix

Visual Studio | Test Explorer | ImageDataLayerTest/0

Test fails. The Output Window shows:

[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from ImageDataLayerTest/0, where TypeParam = struct caffe::CPUDevice<float>
[ RUN      ] ImageDataLayerTest/0.TestResize
<<<<<<<<<<<<<<< End of Output
Warning: Test result file C:\Users\Kevin\AppData\Local\Temp\tmpBCDC.tmp could not be parsed (completely) - your test has probably crashed. Exception message: Root element is missing.
Thread 0 took 00:00:01.7948052
Test execution completed.

If you execute the test from the command line:

Build\x64\Debug\test_all --gtest_filter=ImageDataLayerTest/0*

the test succeeds:

Note: Google Test filter = ImageDataLayerTest/0*
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from ImageDataLayerTest/0, where TypeParam = struct caffe::CPUDevice<float>
[ RUN      ] ImageDataLayerTest/0.TestRead
[       OK ] ImageDataLayerTest/0.TestRead (575 ms)
[ RUN      ] ImageDataLayerTest/0.TestResize
[       OK ] ImageDataLayerTest/0.TestResize (277 ms)
[ RUN      ] ImageDataLayerTest/0.TestReshape
[       OK ] ImageDataLayerTest/0.TestReshape (95 ms)
[ RUN      ] ImageDataLayerTest/0.TestShuffle
[       OK ] ImageDataLayerTest/0.TestShuffle (276 ms)
[----------] 4 tests from ImageDataLayerTest/0 (1230 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test case ran. (1241 ms total)
[  PASSED  ] 4 tests.

Thank you very much.

[jgefele]

Hi Kevin,

can you please post the exact command line used by GoogleTestAdapter? You should find it in the test output window.

The reason I'm asking is the different output / test order in the two test logs above:
GoogleTestAdapter begins with [ RUN ] ImageDataLayerTest/0.TestResize
while your manual run begins with [ RUN ] ImageDataLayerTest/0.TestRead

So I suspect you're using slightly different command line parameters than GoogleTestAdapter. Please try the exact command line parameters used by GoogleTestAdapter manually. If this crashes, too, we have found an explanation.

Another thing to try is Test > Test Settings > Default Processor Architecture > x64.

If none of the above helps, we'll have to further investigate. I won't have time for this before next week. If you want to have a look in the meantime you can to the following:

• Debug (instead of just "run") the test - with a little bit of luck, VS will stop as
  the executable crashes and give you a chance to investigate the cause.
• Otherwise, put a breakpoint at the beginning of the
  ImageDataLayerTest/0.TestResize test (you should be able to navigate
  to that test by clicking on its link in the test explorer)
• Debug the test again
• When your breakpoint is hit, step through the code to identify the
  cause of the test crash

Maybe thge exact cause for the crash gives some hint to what's going wrong.

Cheers,
Jonas

[mwinterb]

Kevin,

From the command line, if you change directories into the Build\x64\Debug directory and then run test_all, do the tests pass?

[csoltenborn]

Hi Kevin,

have you been able to solve your problem? If this is the case, please share your findings, such that we and others can learn from it... Thanks in advance!

And if your problem is still existing: Please let us know if you had the time to try the suggestions provided by @jgefele and @mwinterb (thanks, btw!), and (of course) the respective results.

[KevinLucidyne]

Output window shows the command:
C:\Users\Kevin\dev\IPSight\SegNet\caffe-segnet\Build\x64\Debug\test_all.exe --gtest_output="xml:C:\Users\Kevin\AppData\Local\Temp\tmpFBB2.tmp" --gtest_catch_exceptions=1 --gtest_break_on_failure=1 --gtest_shuffle --gtest_filter=ImageDataLayerTest/0.*:

Running this command in the command prompt shows:

c:\Users\Kevin\dev\IPSight\SegNet\Models\Training>C:\Users\Kevin\dev\IPSight\SegNet\caffe-segnet\Build\x64\Debug\test_all.exe --gtest_output="xml:C:\Users\Kevin\AppData\Local\Temp\tmpFBB2.tmp" --gtest_catch_exceptions=1 --gtest_break_on_failure=1 --gtest_shuffle --gtest_filter=ImageDataLayerTest/0.:
Cuda number of devices: 1
Current device id: 0
Current device name: Quadro M1000M
Note: Google Test filter = ImageDataLayerTest/0.:
Note: Randomizing tests' orders with a seed of 42776 .
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from ImageDataLayerTest/0, where TypeParam = struct caffe::CPUDevice
[ RUN ] ImageDataLayerTest/0.TestResize
E0713 09:12:32.609050 11072 io.cpp:90] Could not open or find file examples/images/cat.jpg
F0713 09:12:32.613530 11072 image_data_layer.cpp:67] Check failed: cv_img.data Could not load examples/images/cat.jpg
*** Check failure stack trace: ***

So problem was a file path problem in one sense but it reveals another problem namely that this error in gtest wasn't picked up by GTA but instead was the error:

Warning: Test result file C:\Users\Kevin\AppData\Local\Temp\tmpBCDC.tmp could not be parsed (completely) - your test has probably crashed. Exception message: Root element is missing.

In other words, the test failed as it should but it didn't give the correct error message.

[csoltenborn]

Hi Kevin,

thanks for the information. The warning GTA produced is correct, since the crashing test didn't allow Google Test to produce a proper test result file - that's what the warning is about. It's only a warning, since we also parse the output produced by the test executable. In fact, the crashing test should have been marked as such in test explorer, and the tests which could not be run because the former one crashed should have pointed to the crashing test.

However, this has indeed uncovered a little bug in GTA: If the crashing test produces some output before crashing (as in your case), that output doesn't end up at the test's Message field in the test explorer (but you can still see that output in the test console if you have enabled Print test output, as you apparently have). We will fix this in the next couple of days...

One final note: I noticed that you are running your tests with shuffling enabled. Note that the main use case for this is to check the quality of your test suite: It is best practice to write unit tests in a way that each test can be executed in total isolation, and thus, they should be executable in any order. The Shuffle tests option helps to find test cases which do not fulfill this requirement. Therefore, my recommendation is to only enable this option if needed.