catkin_add_nosetests fail to run on Trusty in certain configurations due to nosetest bug

[mikeferguson]

This is actually an upstream bug introduced into nosetest 1.3.1 (which ships with Trusty), see nose-devs/nose#779 and the fix in nose-devs/nose@ac9cc08. It appears this doesn't break the "catkin_make run_tests" use case, but it does appear to break https://github.com/ros-infrastructure/jenkins_scripts/blob/master/test_repositories.py#L182 (I'm not sure, but I think this is still how devel jobs are run, right?). Basically, the failure looks like this:

-- run_tests.py: execute commands
  /usr/bin/cmake -E make_directory ../test_results/ubr_drivers
  /usr/bin/nosetests-2.7 -P --process-timeout=60 --where=/home/fergs/tmp/fix-tests/src/test_package/test/python --with-xunit --xunit-file=../test_results/test_package/nosetests-python.xml
.....
Traceback (most recent call last):
  File "/usr/bin/nosetests-2.7", line 9, in <module>
    load_entry_point('nose==1.3.1', 'console_scripts', 'nosetests-2.7')()
  File "/usr/lib/python2.7/dist-packages/nose/core.py", line 121, in __init__
    **extra_args)
  File "/usr/lib/python2.7/unittest/main.py", line 95, in __init__
    self.runTests()
  File "/usr/lib/python2.7/dist-packages/nose/core.py", line 207, in runTests
    result = self.testRunner.run(self.test)
  File "/usr/lib/python2.7/dist-packages/nose/core.py", line 66, in run
    result.printErrors()
  File "/usr/lib/python2.7/dist-packages/nose/result.py", line 110, in printErrors
    self.config.plugins.report(self.stream)
  File "/usr/lib/python2.7/dist-packages/nose/plugins/manager.py", line 99, in __call__
    return self.call(*arg, **kw)
  File "/usr/lib/python2.7/dist-packages/nose/plugins/manager.py", line 167, in simple
    result = meth(*arg, **kw)
  File "/usr/lib/python2.7/dist-packages/nose/plugins/xunit.py", line 199, in report
    self.encoding, 'replace')
  File "/usr/lib/python2.7/codecs.py", line 878, in open
    file = __builtin__.open(filename, mode, buffering)
IOError: [Errno 2] No such file or directory: '../test_results/test_package/nosetests-python.xml'

Hopefully we can actually get this patched upstream in Ubuntu, but I wanted to open a ticket because it took me a while to track this down. Also, I have a somewhat hacky patch that replaces the xunit-file path with an absolute one (https://github.com/mikeferguson/catkin/commit/62e5f68a5def68f6ca5dbdb4267abbffa1513531) that we could use if an upstream fix isn't possible. This doesn't change the catkin_make use case results at all from what I can see, but does make the test job style use case work.

[dirk-thomas]

catkin uses the CATKIN_TEST_RESULTS_DIR directory for nosetest xunit results (

catkin/cmake/test/nosetests.cmake

Lines 72 to 79 in 642af1e

	set(output_path ${CATKIN_TEST_RESULTS_DIR}/${PROJECT_NAME})
	set(cmd "${CMAKE_COMMAND} -E make_directory ${output_path}")
	if(IS_DIRECTORY ${_path_name})
	set(tests "--where=${_path_name}")
	else()
	set(tests "${_path_name}")
	endif()
	set(cmd ${cmd} "${NOSETESTS} -P --process-timeout=${_nose_TIMEOUT} ${tests} --with-xunit --xunit-file=${output_path}/nosetests-${output_file_name}.xml${_covarg}")

) which should always be an absolute path (when using the default value).

But the buildfarm passes in a custom test result directory (https://github.com/ros-infrastructure/jenkins_scripts/blob/master/test_repositories.py#L163) which might expose the bug you referenced. Can you please point me to a specific job failing on Jenkins?

[mikeferguson]

@dirk-thomas There aren't that many devel jobs for indigo, as most of the hydro-devel jobs that I know are using nosetests aren't running indigo-devel jobs right now (which is probably why this hasn't been noticed before) -- most of the python tests end up being rostests in the core ROS packages. I came across this on a private buildbot-ros installation, which uses the exact same logic as test_repositories, and it was a real bear to track down, so I figured I would pass this along as a heads up.

Your commit in nosetests_abs_xunit_file fixes the issue for me and appears to probably be a proper work around for this issue, I've also opened a ticket upstream https://bugs.launchpad.net/ubuntu/+source/nose/+bug/1330932 but I'm not keen on doing the backport myself, or how long it will probably take to actually get merged and released.

In the meantime, I've already patched buildbot-ros to use absolute paths only (mikeferguson/buildbot-ros@fe2901e), and if your nosetests_abs_xunit_file fix cannot be merged, you could probably do similar to test_repositories.py if it becomes a problem.

[dirk-thomas]

I just merged it but it might take some weeks until the next catkin patch release to Indigo. If you are fine until then I won't touch jenkins_scripts. And if it become more a pressuring problem on the farm I will do an immediate release of catkin.

Thanks for digging into it and filling such a detailed ticket!

[mikeferguson]

@dirk-thomas Thanks for finding the catkin patch -- I was trying to do something similar last night with get_filename_component and just didn't quite get it right.

I think we're probably fine for now, I've patched buildbot on my end, and I actually don't find any failing tests on the buildfarm right now since there are so few devel jobs.

[mikeferguson]

@dirk-thomas Just a note, I finally found a package that has a plain nosetest and is running devel job under indigo: xacro (http://jenkins.ros.org/job/devel-indigo-xacro/). There is basically no output in the devel job, but if you look into the package source, there is only a single nosetest.

[dirk-thomas]

That is neither nosetests nor our Jenkins fault. When you checkout the repo and invoke nosetests test it return with Ran 0 tests which will result in Jenkins reporting a failure with None of the test reports contained any result. I just filled that bug upstream: ros/xacro#34