contributing.rst: add testsuite tips

[SteVwonder]

• How to run a single test within the test harness: cd t && make check TESTS=t0000-testname.t
• How to get extra output from the tests: debug=t FLUX_TESTS_LOGFILE=t make check
• Running style checks and formatters: cd $REPO_ROOT && ./scripts/pylint && ./scripts/format
• You can use $REPO_ROOT/scripts/check-format as a git pre-commit hook

Any other useful tips to include?

[grondo]

How to run a single test within the test harness: cd t && make check TESTS=t0000-testname.t

It is much easier to directly run sharness tests since the tap-driver.sh doesn't eat all the debug output:

$ cd t && ./t0000-testname.t -d -v

Python tests can be directly run, but to get the correct PYTHONPATH for your build dir run under flux(1):

$ cd t && ../src/cmd/flux ./python/t000-test.py

or

$ src/cmd/flux start
$ cd t && ./python/t000-testname.py

Running under flux(1) will also sometimes work to allow sub-project tests to run, e.g. in flux-sched.

[grondo]

• Use the make check-prep target at the top-level build directory of flux-core to make everything necessary for running tests, but without running any of the tests under ./t. This is useful if you then want to run only a specific test, or run all tests as Flux jobs, e.g. cd t && flux mini bulksubmit --watch --progress ./{} ::: t[0-9]*.t python/t*.py lua/*.t

[grondo]

• Running all tests in flux-core takes some time. If you are developing a feature and only a few tests are failing make recheck can be useful, it will only re-run failing tests. This makes for a more efficient develop-test-debug cycle.

[grondo]

• Many sharness tests re-run themselves under a Flux instance using the test_under_flux command. For these tests you can set FLUX_TEST_VALGRIND=t and all flux-brokers in the test instance will be run under valgrind for memory debugging. (This is also useful to tease out race conditions in your test code), e.g.:

  $ FLUX_TEST_VALGRIND=t ./t000-testname.t -d -v
  sharness: loading extensions from /home/grondo/git/f.git/t/sharness.d/01-setup.sh
  sharness: loading extensions from /home/grondo/git/f.git/t/sharness.d/flux-sharness.sh
  ==5906== Memcheck, a memory error detector
  ==5906== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
  ==5906== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
  ==5906== Command: /home/grondo/git/f.git/src/broker/.libs/flux-broker --setattr=rundir=/tmp/flux-5803- 
  

[chu11]

dunno if flux-sched specific test techniques want to be detailed separately or in the same area? I think you have to do the:

$ src/cmd/flux start
$ cd t && ./t0000-foo.t

or the

make check TESTS=t0000-testname.t

style

[SteVwonder]

Less of a testsuite tip and more of a test debugging tip, but you can set FLUX_HANDLE_TRACE in your environment, and Flux will dump all of the messages on the overlay to stderr. If you only want to see the messages for a single handle, you can pass FLUX_O_TRACE as a flag to flux_open.