Set location of coverage file in Travis

[kinow]

Adding a sed command to our .travis.yml to use an absolute directory for the coverage data file. This way when processes are forked, they will use the same location, instead of a location relative to their working directory.

This should increase the coverage of the project, and cover files such as daemonize.py.

Bruno

[kinow]

It looks like the daemonize.py is still not covered in the coverage report for this branch/PR 😭

[oliver-sanders]

Dammit!

[kinow]

😠 tried moving the .coveragerc under /tmp/ and then specify it with coverage run --rcfile... but I guess there are still relative folders... coverage report still giving 0 runs for the daemonize.py file

[kinow]

@oliver-sanders and @matthewrmshin,

I followed my steps from yesterday again this afternoon. And succeeded in running a local coverage command that included scheduler.py and daemonize.py.

But then I realized there is something funny still happening when it runs with Travis.

I've succeeded in getting the coverage for those files, when I ran from command line something similar to:

$ export TRAVIS_BUILD_DIR=$(pwd -P)
$ sed -i -e "s|data_file=.coverage|data_file=${TRAVIS_BUILD_DIR}/.coverage|g" .coveragerc
$ export COVERAGE_PROCESS_START="${TRAVIS_BUILD_DIR}/.coveragerc"
$ export PYTHONPATH=$(pwd -P)/.travis
$ cylc register upstream tests/suite-state/upstream
$ cylc register polling tests/suite-state/polling
$ cylc run upstream
$ cylc run --no-detach polling --set UPSTREAM=upstream
$ coverage combine --append
$ coverage html

Note that I don't even need to run coverage run. It produces the right values in the reports.

However, when instead of registering the suites and running them, I call the test-battery command instead, it doesn't produce the right values:

$ export TRAVIS_BUILD_DIR=$(pwd -P)
$ sed -i -e "s|data_file=.coverage|data_file=${TRAVIS_BUILD_DIR}/.coverage|g" .coveragerc
$ export COVERAGE_PROCESS_START="${TRAVIS_BUILD_DIR}/.coveragerc"
$ export PYTHONPATH=$(pwd -P)/.travis
$ cylc test-battery tests/suite-state/01-polling.t
$ coverage combine --append
$ coverage html

If I call prove, it also doesn't work.

$ export TRAVIS_BUILD_DIR=$(pwd -P)
$ sed -i -e "s|data_file=.coverage|data_file=${TRAVIS_BUILD_DIR}/.coverage|g" .coveragerc
$ export COVERAGE_PROCESS_START="${TRAVIS_BUILD_DIR}/.coveragerc"
$ export PYTHONPATH=$(pwd -P)/.travis
$ export CYLC_TEST_RUN_GENERIC=${CYLC_TEST_RUN_GENERIC:-true}
$ export CYLC_TEST_RUN_PLATFORM=${CYLC_TEST_RUN_PLATFORM:-true}
$ export CYLC_TEST_SKIP=${CYLC_TEST_SKIP:-}
$ export CYLC_TEST_IS_GENERIC=true
$ export CYLC_TEST_TIME_INIT="$(date -u +'%Y%m%dT%H%M%SZ')"
$ prove -s -r "./tests/suite-state/01-polling.t"
$ coverage combine --append
$ coverage html

Finally, it does work if I alter 01-polling.t, and instead of suite_run_ok $TEST_NAME cylc run --reference-test --debug --no-detach --set UPSTREAM=$UPSTREAM $SUITE_NAME, I remove the suite_run_ok $TEST_NAME part. So the following works.

$ export TRAVIS_BUILD_DIR=$(pwd -P)
$ sed -i -e "s|data_file=.coverage|data_file=${TRAVIS_BUILD_DIR}/.coverage|g" .coveragerc
$ export COVERAGE_PROCESS_START="${TRAVIS_BUILD_DIR}/.coveragerc"
$ export PYTHONPATH=$(pwd -P)/.travis
$ export CYLC_TEST_RUN_GENERIC=${CYLC_TEST_RUN_GENERIC:-true}
$ export CYLC_TEST_RUN_PLATFORM=${CYLC_TEST_RUN_PLATFORM:-true}
$ export CYLC_TEST_SKIP=${CYLC_TEST_SKIP:-}
$ export CYLC_TEST_IS_GENERIC=true
$ export CYLC_TEST_TIME_INIT="$(date -u +'%Y%m%dT%H%M%SZ')"
# EDIT 01-polling, leaving just cylc run --reference-test --debug --no-detach --set UPSTREAM=$UPSTREAM $SUITE_NAME
$ prove -s -r "./tests/suite-state/01-polling.t"
$ coverage combine --append
$ coverage html

The test fails due to the number of tests, but still produces the right coverage.

I had a look at the function in test-headers... but couldn't find anything that would initiate a shell without the environment variables necessary for coverage (even altered it a bit to print env | sort, and the variables were there). (EDIT: funny, it worked in NIWA, but just tried all examples at home, and this last one also failed to report the right coverage).

So really awkward that running through cylc test-battery it simply doesn't follow what is in .coveragerc file 😞

[kinow]

Ra! Enabled the trace option for debugging. Turns out the source directories were not matching. So when the process is forked, it cannot find the coverage RC file, but also it tries to load the source folders (./lib/ and ./bin/) relative to the working directory that may not match Cylc's git checked out directory).

Fixed by adding two more sed expressions. It looks like it resolves all relative folders to absolute folders in the configuration file. So hopefully no more surprises when running tests in Travis CI.

And, good news! Coverage went up to 83.99%! 🎉 Might have another look at what is not being tested and see if there's anything to keep an eye on during the Python3 migration. But the coverage of the existing functional tests is really impressive. Well done!! 👏 👏

This pull request is ready for review.

[matthewrmshin]

We are not so bad after all!

[hjoliver]

Brilliant news, made my day 😁 👏

[hjoliver]

(I was a little surprised that our tests didn't give better coverage than seemed to be the case at first... so nice to find our intuition wasn't too far off after all!)

[oliver-sanders]

84% seems a bit more realistic to me!