-
Notifications
You must be signed in to change notification settings - Fork 162
Testing methods and targets
This page describes the different test targets I would like to introduce. It is important that the make targets for tests are exactly the thing that run on travis, so that contributors who submit pull requests can run these tests themselves and get consistent results.
A test target will be a combination of a suite and a level, where we currently
define two suites: core and dist, and three levels: quick, full, and intense.
Running a test should in the end be as simple as
# make test-core-full
The core tests are only permitted to test the core system and only
rely on core functionality, and NeededPackages. Tests in this category can
only rely on GAP as is installed after cloning the git repository, building
and executing make bootstrap-pkg-minimal.
To test non-reliance on packages, the tests will be run with the -A option.
to test that packages loaded by default do not break core tests, the tests
will be run without the -A option, but after make bootstrap-pkg-full.
The distribution tests test GAP as it is distributed, that is with
packages in the bootstrap-pkg-full archive. Tests here can rely on
package functionality that is available. Test authors should carefully
consider moving tests that rely on package functionality to the package
in question.
These tests will be run with with and without -A option as well.
Remark: in Jenkins, we will also run them with ALL packages loaded.
We assume reasonably modern hardware in a laptop (2 cores/4 threads, some RAM)
A single quick test should finish within seconds at most, and a
single quick test file should not take longer than a minute. The
whole make-test-*-quick target should take no more than several minutes.
The rationale is that people should be able to run this test target
and not be bothered by it.
A single line in a full testfile should finish within minutes, a
full test file should not take longer than ten minutes. The whole
make-test-*-full target should take less than an hour. This should
still be quick enough to be run before submitting a pull
request that changes some major bits.
An intense test can basically take as long as it wants to, and exercise the system in ways that are not possible with shorter tests. These tests will be run on a regular basis (like every week), but not as part of CI
| command | file | target | |
|---|---|---|---|
| quick | < 2 seconds | < 1 minute | 3-5 minutes? |
| full | < 1 minute | < 5 minutes | 40-50 minutes? |
| intense | any | any | any |