Path errors when running make test

[skylian]

After mkdir build && cd build and cmake .., running make test under the folder build will fail on the following tests:

24 - test_PyDataProvider (Failed)
26 - test_RecurrentGradientMachine (Failed)
27 - test_NetworkCompare (Failed)
28 - test_PyDataProvider2 (Failed)
32 - test_Prediction (Failed)
33 - test_Compare (Failed)
34 - test_Trainer (Failed)
35 - test_TrainerOnePass (Failed)
36 - test_CompareTwoNets (Failed)
37 - test_CompareTwoOpts (Failed)
38 - test_CompareSparse (Failed)
39 - test_recurrent_machine_generation (Failed)
40 - test_PyDataProviderWrapper (Failed)
41 - test_config_parser (Failed)
42 - test_swig_api (Failed)
43 - layers_test (Failed)

The causes are all about path errors. For example, on line 39 of test_PyDataProvider.cpp , the required file is not under build/paddle/gserver/tests, but under paddle/gserver/tests.

Copying all necessary files into build folder will solve the problems.

[yu239-zz]

Did you cmake again after git pull? Please use ctest -V to print out the error message.

[luotao1]

Do you make install the paddle? You can use make install in your build folder before make test.

[skylian]

@yu239 Yes. That was what I did. I got the error messages. They are all path errors, like the following for test_PyDataProvider

I1106 21:53:20.013489 31779 Util.cpp:155] commandline: ./paddle/gserver/tests/test_PyDataProvider
I1106 21:53:20.910562 31779 hl_dso_loader.cc:51] Try to find cuda library: libcublas.so from default system path.
I1106 21:53:20.973400 31779 hl_dso_loader.cc:51] Try to find cuda library: libcurand.so from default system path.
I1106 21:53:20.978911 31779 hl_dso_loader.cc:51] Try to find cuda library: libcudnn.so from default system path.
I1106 21:53:21.378783 31779 Util.cpp:130] Calling runInitFunctions
I1106 21:53:21.378943 31779 Util.cpp:143] Call runInitFunctions done.
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from PyDataProvider
[ RUN ] PyDataProvider.py_fill_slots
F1106 21:53:21.384443 31779 Util.cpp:298] Check failed: is Fail to open gserver/tests/pyDataProvider/pyDataProviderList
*** Check failure stack trace: ***
@ 0x7fba30c81daa (unknown)
@ 0x7fba30c81ce4 (unknown)
@ 0x7fba30c816e6 (unknown)
@ 0x7fba30c84687 (unknown)
@ 0x8043ac paddle::loadFileList()
@ 0x553208 paddle::PyDataProvider::PyDataProvider()
@ 0x55372d _ZNSt17_Function_handlerIFPN6paddle12DataProviderENS0_10DataConfigENS0_11ModelConfigEbEZNKS0_UlvE_clEvEUlS3_S4_bE_E9_M_invokeERKSt9_Any_dataS3_S4_b
@ 0x56596f paddle::DataProvider::create()
@ 0x54617b PyDataProvider_py_fill_slots_Test::TestBody()
@ 0x8b5aca testing::internal::HandleSehExceptionsInMethodIfSupported<>()
@ 0x8b0b10 testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x89dd03 testing::Test::Run()
@ 0x89e47e testing::TestInfo::Run()
@ 0x89ea3e testing::TestCase::Run()
@ 0x8a3698 testing::internal::UnitTestImpl::RunAllTests()
@ 0x8b7014 testing::internal::HandleSehExceptionsInMethodIfSupported<>()
@ 0x8b1abc testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x8a2495 testing::UnitTest::Run()
@ 0x538e78 main
@ 0x7fba2fe8df45 (unknown)
@ 0x543105 (unknown)
@ (nil) (unknown)
Aborted (core dumped)

[skylian]

@luotao1 Yes. I ran make install before running make test. Didn't find the places in cmake files (e.g. here) where the necessary files were copied or linked to the build folder.

[hedaoyuan]

@skylian
Check ./paddle/gserver/tests/CTestTestfile.cmake file, take a look at how the test_PyDataProvider is add_test. And take a look at whether the paddle/.set_python_path.sh file exists.

[luotao1]

Maybe the problem of python path. You can use pip uninstall paddle, and make install again.

[skylian]

@hedaoyuan @luotao1 The problem is in PYTHONPATH, not path errors that I thought were.

I pip install paddle before running tests and got error messages from paddle/.set_python_path.sh saying "paddle package is already in your PYTHONPATH. But unittest need a clean environment. Please uninstall paddle package before start unittest. Try to pip uninstall paddle". from line 25 of .set_python_path.sh. Found it mentioned in #345. Removing that line or pip uninstall paddle before testing solve the problem.

What is the purpose of the line?

[reyoung]

@skylian 如果简单的说，原因就是 unittest need a clean environment.

复杂一点说，如果paddle包已经在python的site-packages里面了，那么后面单元测试时，使用的paddle包，就是site-packages里面的python包，而不是源码目录里/python目录下的python包了。即使设置了PYTHONPATH到/python，因为python的搜索路径似乎优先已经安装的python包。

这会导致，在运行单元测试的时候，使用了一个旧版本的python包，而没有测试到代码中实际修改的python包。

[reyoung]

@luotao1 也许我们可以把这个事情放到FAQ里面。

[luotao1]

@reyoung 我可以把这个整理下放到FAQ中，这个问题已经出现了很多次了

[reyoung]

cool.
FAQ文档的地址是 https://github.com/baidu/Paddle/blob/develop/doc_cn/faq/index.rst

[reyoung]

@luotao1 不过我们可以先把要写的东西放到issue上，加上help wanted标签。就像这样
#373

万一团队外面的人想要帮着写一下，可以让他们贡献这些。

[luotao1]

@reyoung got it!