make "make runtest" independant on source tree

[EricDeveaud]

Hi,

all our softwares are available thru a NFS read-only file system on the cluster.
so caffe-rc3 is
the build machine (the only one with rw premissions) does not hold gpu card

so running make runtest is not possible in this machine.
after successful make test on the build machine and installation of the distribute dir stuff, in order to validate our caffe installation, I grabbed the whole caffe-rc3/.build_release/test/ to a node holding gpus
then issued the following cmd. (stolen from make runtest rule)
'''
./test/test_all.testbin 0 --gtest_shuffle
'''

and got 2 test failures:
test/test_gradient_based_solver.testbin
test/test_hdf5data_layer.testbin

both fail because of the lack of missing data files

'''
[ RUN ] HDF5OutputLayerTest/3.TestForward
HDF5-DIAG: Error detected in HDF5 (1.8.14) thread 0:
#000: H5F.c line 604 in H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#1: H5Fint.c line 990 in H5F_open(): unable to open file: time = Fri Jul 22 12:57:36 2016
, name = 'src/caffe/test/test_data/sample_data.h5', tent_flags = 0
major: File accessibilty
minor: Unable to open file
#2: H5FD.c line 992 in H5FD_open(): open failed
major: Virtual File Layer
minor: Unable to initialize object
#3: H5FDsec2.c line 343 in H5FD_sec2_open(): unable to open file: name = 'src/caffe/test/test_data/sample_data.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0
major: File accessibilty
minor: Unable to open file
src/caffe/test/test_hdf5_output_layer.cpp:78: Failure
Expected: (file_id) >= (0), actual: -1 vs 0
Failed to open HDF5 filesrc/caffe/test/test_data/sample_data.h5
[ FAILED ] HDF5OutputLayerTest/3.TestForward, where TypeParam = caffe::GPUDevice (2 ms)
'''
and

'''
[ RUN ] NesterovSolverTest/2.TestNesterovLeastSquaresUpdateWithWeightDecayMultiIter
F0722 12:57:36.373814 6208 hdf5_data_layer.cpp:88] Failed to open source file: src/caffe/test/test_data/solver_data_list.txt
*** Check failure stack trace: ***
@ 0x7f350f3d2d0d google::LogMessage::Fail()
@ 0x7f350f3d80ba google::LogMessage::SendToLog()
@ 0x7f350f3d4de0 google::LogMessage::Flush()
@ 0x7f350f3d50cd google::LogMessageFatal::~LogMessageFatal()
@ 0x7f3508b44423 caffe::HDF5DataLayer<>::LayerSetUp()
@ 0x7f3508ba48ea caffe::Net<>::Init()
@ 0x7f3508ba6d05 caffe::Net<>::Net()
@ 0x7f3508b6154a caffe::Solver<>::InitTrainNet()
@ 0x7f3508b61c34 caffe::Solver<>::Init()
@ 0x7f3508b62168 caffe::Solver<>::Solver()
@ 0x6ce118 caffe::NesterovSolverTest<>::InitSolver()
@ 0x6c7ad8 caffe::GradientBasedSolverTest<>::InitSolverFromProtoString()
@ 0x6f610a caffe::GradientBasedSolverTest<>::RunLeastSquaresSolver()
@ 0x6f99fe caffe::GradientBasedSolverTest<>::TestLeastSquaresUpdate()
@ 0x6f9fab caffe::NesterovSolverTest_TestNesterovLeastSquaresUpdateWithWeightDecayMultiIter_Test<>::TestBody()
@ 0x901c1d testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x8f3f21 testing::Test::Run()
@ 0x8f4007 testing::TestInfo::Run()
@ 0x8f4147 testing::TestCase::Run()
@ 0x8f904f testing::internal::UnitTestImpl::RunAllTests()
@ 0x9017cd testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x8f354a testing::UnitTest::Run()
@ 0x50a8ff main
@ 0x3fe321ed1d (unknown)
@ 0x50a629 (unknown)
zsh: abort (core dumped) LD_LIBRARY_PATH= ./test/test_all.testbin 0 --gtest_shuffle
'''

may I suggest that test_data will be embeded in the test folder in order to be independant of the src tree.

NB: running caffe/rc3 cuda/7.5.18 cudnn/v4 on CentOS release 6.8 (Final)
caffe was built versus gflags/2.1.2 atlas/3.10.2 protobuf/2.6.1 boost/1_56_0 hdf5/1.8.14 glog/0.3.4
with 2 Mk40 gpu cards

best regards

Eric

[EricDeveaud]

everything went smoothly once I created src/caffe/test in the same directory as test and transfered test_data folder to it.

[seanbell]

This sounds like a somewhat unique situation. Yes, you need to be able to write to the directory in order to build it. There is no way around that. I would suggest making a copy of the code and then building it there.

I am closing this since it's more of a usage/installation question.

From https://github.com/BVLC/caffe/blob/master/CONTRIBUTING.md:

Please do not post usage, installation, or modeling questions, or other requests for help to Issues.
Use the caffe-users list instead. This helps developers maintain a clear, uncluttered, and efficient view of the state of Caffe.