Open Test in Test Explorer doesn't work

[KevinLucidyne]

Same setup as Issue #57. When a test fails or completes successfully, F12 or "Open Test" do not open the test source file.

[csoltenborn]

Please be more specific, since I do neither know the F12 nor the Open test functionality, I must admit :-)

Again, two cases here:

• If you click the "Source:" link in the test explorer, you should end up at the source location of the according test
• Clicking on stack traces of failed tests has issues at the moment (see Links to test failures do not always work #65)

[KevinLucidyne]

Using setup in Issue #57 running test in Release mode for AccuracyLayerTest/0.TestForwardCPU.

Result in bottom pane of Test Explorer:

AccuracyLayerTest/0.TestForwardCPU
Source: no source available
Test Passed - blah blah blah

Right click on AccuracyLayerTest/0.TestForwardCPU in top pane of Test Explorer. Context menu shows:

Run Selected Tests
...
Open Test F12

Clicking on Open Test does nothing.

Source for this file is available as I can see when I go to the source code of the test itself in my solution.

Thanks!

[csoltenborn]

Sorry, I never noticed that entry in the tests' context menu... The good news is that it is working in general (for me) - the actual issue is that in your case, GTA is not able to find the source. Is this happening for all tests of a test executable? Anyways, please check out our trouble shooting section as a first step...

[KevinLucidyne]

Yes, it happens for all the tests in the test executable. The troubleshooting section didn't help.

[csoltenborn]

What's the size of the test executable and the belonging pdb file?

[KevinLucidyne]

07/20/2016 11:12 AM 32,252,928 test_all.exe
07/20/2016 09:28 AM 77,711 test_all.exe.gta.testdurations
07/20/2016 11:12 AM 145,519,184 test_all.ilk
07/20/2016 11:12 AM 93,564,928 test_all.pdb
5 File(s) 271,414,751 bytes

[csoltenborn]

Please download and install the vsix file produced by this build (which adds some debug output), enable Print test output and Print debug info, and upload the complete output GTA produces in the test output window.

[csoltenborn]

Thanks! It indeed looks like this is some bug with type-parameterized tests... We will look into it.

Two remarks:

• It noticed that in the code you are working on, the test files contain tests called Setup. This is a clear indication that you should not use random execution order on those tests (most likely, a test called Setup shall be executed before the other tests of its "group")
• With "uploading the complete output" I meant uploading in the literal way, i.e. zipping a file containing that output and uploading it as a file ;-) - would you mind to edit your comments that way? Makes this issue much less cluttered and easier for us to investigate the logs, since we can work in our favorite editors... thanks in advance. And maybe you can rename the issue in the line of "Source file locations of type-parameterized tests are not always found". I could do the latter for myself, but don't want to be rude...

[KevinLucidyne]

Sorry, don't have that output anymore. I'll remember to zip it in the
future. Feel free to delete the comments or edit the entry in any way if
you like. This is your thing. I'm just using your thing for doing other
things, which are my main priority. :-)

Kind regards,
Kevin

On Thu, Jul 21, 2016 at 10:09 PM, Christian Soltenborn <
notifications@github.com> wrote:

Thanks! It indeed looks like this is some bug with type-parameterized
tests... We will look into it.

Two remarks:

• It noticed that in the code you are working on, the test files
  contain tests called Setup. This is a clear indication that you should not
  use random execution order on those tests (most likely, a test called Setup
  shall be executed before the other tests of its "group")
• With "uploading the complete output" I meant uploading in the
  literal way, i.e. zipping a file containing that output and uploading it as
  a file ;-) - would you mind to edit your comments that way? Makes this
  issue much less cluttered and easier for us to investigate the logs, since
  we can work in our favorite editors... thanks in advance. And maybe you can
  rename the issue in the line of "Source file locations of
  type-parameterized tests are not always found". I could do the latter for
  myself, but don't want to be rude...

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
#68 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ASmDeTuchjzQ56GHq1b7VYHgKXb6Ckerks5qYFCTgaJpZM4JP_8g
.

Kevin Johnsrude
Senior Software Engineer
Lucidyne Technologies, Inc.
155 SW Madison Ave.
Corvallis, Oregon 97330
541 753 5111
www.Lucidyne.com

[csoltenborn]

Right, but you could (as I could) pull the changes out of the comments while editing them, put them in a text file, zip that file, and upload it as suggested. Don't get me wrong: I don't really care, but I'm certainly not doing the work for you ;-) If people look at the quality of the issues you filed, they will probably have to smile if they see this one... And if you are using other things to do your things, and if you need them improved to be able to do your things, you better try to be as supportive as possible! Not at all meant as a threat, just as a recommendation...

[jgefele]

Caffe is using a different (older?) version of gtest.

They are not printing the type parameters as expected:

test_all.exe --gtest_list_tests --gtest_filter=AccuracyLayerTest*TestForwardCPU
AccuracyLayerTest/0.
  TestForwardCPU
AccuracyLayerTest/1.
  TestForwardCPU

instead of

test_all.exe --gtest_list_tests --gtest_filter=AccuracyLayerTest*TestForwardCPU
AccuracyLayerTest/0.  # TypeParam = float
  TestForwardCPU
AccuracyLayerTest/1.  # TypeParam = double
  TestForwardCPU

For this reason, we're not finding the symbols, because we don't know the types "float" and "double".

Found symbol 'caffe::AccuracyLayerTest_TestForwardCPU_Test<float>::TestBody': 1, 1439872, 1015
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:111
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:112
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:113
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:114
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:115
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:119
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:120
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:121
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:122
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:123
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:124
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:125
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:126
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:123
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:129
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:130
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:120
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:133
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:135
Found symbol 'caffe::AccuracyLayerTest_TestForwardCPU_Test<double>::TestBody': 1, 1440896, 1015
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:111
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:112
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:113
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:114
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:115
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:119
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:120
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:121
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:122
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:123
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:124
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:125
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:126
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:123
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:129
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:130
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:120
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:133
   Found source line: c:\users\vagrant\source\repos\caffe\src\caffe\test\test_accuracy_layer.cpp:135

I couldn't find out the version of gtest which caffe is using. But I diffed the source and their version's UnitTestImpl::ListTestsMatchingFilter() is missing the parts that print the TypeParam.

So I guess there is not much we can do with the current way of finding source files. Maybe, instead of searching for "TestSuite_TestName< TypeParam>::TestBody" we could match "TestSuite_TestName<*>::TestBody" and select the first one. They should all point to the same source line anyway...

[csoltenborn]

@KevinLucidyne First of all, thanks for cleaning up this issue - makes handling a bit easier indeed!

I have implemented the suggestions arising from Jonas' analysis - can you give this build a try? Please let us know whether source file locations are now found for the according tests, and (obviously) whether they are correct in case they are found...

[KevinLucidyne]

Loaded new build. Restarted VS2013. Ran AccuracyLayerTest/0 group of tests. Source not found. Test output attached.
Bug68.txt

[KevinLucidyne]

Rather than guessing and tossing possible fixes over the wall to me, how about developing against the problem on your local machines? Set up is easy.

Download and compile this code on VS 2013:
https://github.com/BVLC/caffe/tree/windows

It may even be the same problem on VS2015 but in any case the Community Editions of the VS suite are freely and widely available.

[csoltenborn]

Well, to be honest, I have tried that, but didn't suceed (with VS 2015), and I don't want to mess with VS 2013 on my machine. Which would mean setting up a virtual machine etc., and I don't have the time for that at the moment. However, Jonas and I have a pretty clear idea on what was missing in the version I provided for you, so I will give it one more try in the next couple of days. If that version doesn't work either, we probably have to re-think our strategy :-)

[csoltenborn]

@KevinLucidyne As it turns out, Jonas has already set up your project on his machine, so he can test whether my latest patch indeed fixes the issue. However, I'm not sure when he will time for that, so if you want to give it another shot, please give this build a try...

[KevinLucidyne]

Works! Thanks!