Policies | Testing

Trilinos Test Harness

Contents:

• Overview
• Categories of Tests
• Finding information on CDash

Overview

Trilinos uses CTest and CDash for testing and display of test results. With CTest/CDash, a testing day is from 10 p.m. to 10 p.m. MST. CDash pages for current results can be found below.

Nightly builds test branches that are updated at the same time each day (9 p.m. MST) so that all of the builds for a given testing day test the exact same version of Trilinos. On the CI builds test the live version of the Trilinos develop branch.

Links to main Trilinos CDash dashboard results and query tools:

• Main projects dashboard page for the current day
• Subprojects (packages) dashboard page for Trilinos for the current day [Warning: can be slow to come up]
  • Tip: To get a quick summary for all the packages that had configure, build, or test failures, sort the packages in descending order by holding down the Shift key and then click twice on the column headers Configure: Error, Build: Error, then Test: Fail.
• Subproject (package) dashboard page Trilinos package Teuchos for the current day
  • Tip: To query other packages change 'Teuchos' in the URL to some another package name (e.g. 'Epetra', 'Tpetra', etc.)
• Summary of Trilinos builds for the current day
  • Tip: To get a quick summary for all the build groups that had configure, build, or test failures, sort the build group rows in descending order by holding down the Shift key and then click twice on the column headers Configure: Error, Build: Error, then Test: Fail.
  • Tip: To see the individual package builds/tests, click on the name of the build in the Build Name field.
  • Tip: To see what files got updated, click the number in the Update field.
  • Tip: To exclude bad packages (i.e. packages with build and/or test failures) from the build summaries, click [Show Filters] and then select the filter ('Subprojects', 'exclude', '<BAD_PACKAGE_NAME>').
• Test query page:
• All tests for current day [Warning: very slow to come up]
• Failing tests for current day [Comes up fast]
  • Tip: To see all runs for a given test <TestName>, use filter ('Test Name', 'starts with', '<TestName>')
  • Tip: To query results for <PACKAGEX>, select the filter ('Test Name', 'starts with', 'PACKAGEX_')
  • Tip: To query passing tests, select the filter ('Status', 'is', 'passed') [Note the lower case 'passed')
  • Tip: To query failing tests, select the filter ('Status', 'is', 'failed') [Note the lower case 'failed')
  • Tip: To see tests for the last two weeks, select the filter ('Build Time', 'is after', '2 weeks ago')
• Current coverage results (as of last Saturday)
• Tip: Click on the Build Name for the MPI or Serial build under 'Nightly' and then go to 'Coverage' to the coverage results for each Trilinos package.

Categories of Tests

The different categories of Trilinos testing are:

• Pull-request Testing
• Deprecated : Pre-push Continuous Integration Testing
• Post-push Continuous Integration Testing
• Dev->Master (Develop to Master Branch) Testing
• Nightly Regression Testing
• Performance Testing
• Installation Testing
• Experimental Testing
• Coverage Testing
• Memory Testing
• Scalability Testing

These different categories of testing are described in more detail below.

Pull-request testing: All packages [Pull Request Builds]

When a new pull request is issued to the trilinos/Trilinos repository Trilinos/pulls this system will create jobs on a set of Sandia machines to build and test. The build results for the builds are summarized in the GitHub issue, and are also reported in more detail to CDash in the "Pull Request" group. More detailed information on pull-request testing can be found here.

Deprecated : Pre-push Continuous Integration Testing: Primary Tested Code [Checkin test mailing list]

Previously, before every push to the global Trilinos git repository 'develop' branch, the Trilinos checkin-test-sems.sh script would be used to test all affected Primary Tested (PT) Code. The pull request testing system is replacing this functionality as it tests more platorms and can be readily expanded as needed. The purpose for this type of testing is to a) do a basic smoke test to make sure nothing significant has been broken, and b) provide a consistent basis of comparison across all developers to determine if it is safe to push.

The main purpose of this type of testing is to protect other developers so they can continue their development work after pulling changes from the 'develop' branch. The selection of PT packages is done based on usage by important Trilinos customers (see Trilinos GitHub Issue #410). However,

NOTE: By default, all pushes done through checkin-test-sems.sh [other options] --push get archived to the trilinos-checkin-tests@software.sandia.gov Mailman list. If your pushes are not getting archived to this list please send email to trilinos-framework@software.sandia.gov to have your URL added the list of accepted submitters.

The default test category set by the Trilinos checkin-test-sems.sh script is Trilinos_TEST_CATEGORIES=BASIC which matches to the CATEGORIES option BASIC only (not NIGHTLY) passed into TRIBITS_ADD_TEST() and TRIBITS_ADD_ADVANCED_TEST() in the test CMakeLists.txt files.

Post-push Continuous Integration Testing: Primary Tested Code [Continuous Integration Build]

A Continuous Integration (CI) server is set up that run the build Linux-GCC-4.8.4-MPI_RELEASE_DEBUG_SHARED_PT_OPENMP_CI in a continuous loop looking for updates to the global repository and when it finds them, it enables the affected packages and runs the builds and tests. This build is an exact duplicate of the default build MPI_RELEASE_DEBUG_SHARED_PT_OPENMP_CI which is run by the checkin-test-sems.sh script.

Emails are sent out by CDash to the different package lists PACKAGE-regression@software.sandia.gov when errors are detected. The first build of the day is a build from scratch while every subsequent CI iteration is a rebuild and retesting of only the affected packages. (This greatly speeds up the response time of the CI server compared to always building and testing of of Trilinos at every iteration.)

The main purposes for having a post-push CI server running in addition to pre-push CI testing are to: a) quickly catch errors from sloppy (or absent) checkin testing, b) catch violations of the additive test assumption of branches which can occur when multiple test/pushes overlap, c) catch problems so they can be fixed before the nightly or weekly tests fire off.

Dev-Master Testing: All packages [Pull Request Builds]

The purpose of the dev->master branch testing is to provide additional stability checks beyond pull-request testing for users of the master branch.

The dev->master testing process creates a temporary branch from the develop branch, and issues a pull-request against master. The dev->master set of tests is run against that temporary branch and can be merged if all tests pass.

Nightly Regression Testing: Secondary Tested Code [Nightly Regression Dashboard]

Each night starting at 12 Midnight MST/MDT, a wide range of configurations of Trilinos are tested on a variety of different platforms for Secondary Tested (PT) Code. MPI and SERIAL are varied along with DEBUG and RELEASE. Different compilers are used. Builds are performed with shared or static libraries.

The test category set for nightly testing is Trilinos_TEST_CATEGORIES=NIGHTLY which matches to the CATEGORIES options BASIC or NIGHTLY passed into passed into TRIBITS_ADD_TEST() and TRIBITS_ADD_ADVANCED_TEST() in the test CMakeLists.txt files.

Performance Testing: Secondary Tested Code [Nightly Performance Dashboard]

!!! NOTE: PERFORMANCE TESTING IS CURRENTLY DISABLED !!!

Performance tests are specially designed and specially run tests to assess the serial performance of code. These tests do timing of different types of operations and have strong timing pass/fail test success criteria. If the relative or absolute time limit is violated, then the test fails. These tests must typically be targeted to a specific computer and manually verified on the machine before they can set as nightly tests.

The test category set for performance testing is Trilinos_TEST_CATEGORIES=PERFORMANCE which matches to the CATEGORIES option PERFORMANCE only passed into TRIBITS_ADD_TEST() and TRIBITS_ADD_ADVANCED_TEST() in the test CMakeLists.txt files.

Installation Testing: Primary Tested Code

!!! NOTE: INSTALLATION TESTING IS CURRENTLY DISABLED !!!

Installation testing is currently done on Primary Tested code to test whether or not an installation contains all necessary files. It does this by compiling Trilinos package tests against an installation of Trilinos that was built from a release-like tarball generated from the development branch. This helps ensure that both the installation and the tarball are not missing critical files.

Installation testing can also be run manually so developers can check if recent changes affected the installation, or reproduce issues found in the nightly testing if necessary. To generate a release-like tarball of Trilinos (on a Linux system), configure Trilinos as for a normal build, and then type make package_source. Next, expand the tarball and configure, build, and install Trilinos, using CMAKE_INSTALL_PREFIX to specify the installation directory. Next, using the git clone used to create the release-like tarball, configure Trilinos again, this time setting the variable Trilinos_INSTALLATION_DIR to the the path that CMAKE_INSTALL_PREFIX was set to when installing Trilinos. All other configuration options should be the same for both builds to prevent incompatibilities (e.g. trying to build MPI tests against a serial installation). Finally, type make, and run the tests (using ctest.

The second build will not compile Trilinos libraries, but rather link against the libraries built for the tarball-based build. The header files in the source trees for Trilinos packages will also be ignored in favor of the headers in the installation directory.

The first build can be based directly on a git clone, rather than a release-like tarball, but some errors will only be detected when building from a tarball.

Experimental Testing: Experimental Code [Experimental Dashboard]

The experimental dashboard contains a host of experimental builds where no emails are sent out for failed builds. Some of these experimental builds are run by cron jobs every night. Regularly scheduled builds that are run as experimental builds usually have some issues that result in several failed package builds or test failures and are not ready for "prime time" yet. The goal is to get these builds to a state where they can be moved over to regular "Nightly" builds where emails will be sent out reporting failures.

Other experimental builds are run by individual developers (for running coverage testing for instance). A good way to post experimental builds from a local git repo and build directory is to use the 'dashbaord' target run with make dashboard.

By default, the test category set for experimental testing is Trilinos_TEST_CATEGORIES=NIGHTLY which matches to the CATEGORIES options BASIC or NIGHTLY passed into TRIBITS_ADD_TEST() and TRIBITS_ADD_ADVANCED_TEST() in the test CMakeLists.txt files.

Coverage Testing: Secondary Tested Code [Current coverage results (as of last Saturday)]

!!!NOTE: REGULAR COVERAGE TESTING IS CURRENTLY DISABLED !!!

Every Saturday morning, a set of coverage tests is run after the other nightly tests are finished. An MPI and a SERIAL build are performed where the coverage for each package is accumulated after running each of the package's test suite.

Note that this coverage will typically be lower than the actual coverage after running the rest the downstream package tests that test more functionality. Currently, we are not accumulated these higher coverage numbers. Trilinos needs to strive to improve the coverage of the native packages test suites for each and every package.

The test category set for coverage testing is Trilinos_TEST_CATEGORIES=NIGHTLY which matches to the CATEGORIES options BASIC or NIGHTLY passed into TRIBITS_ADD_TEST() and TRIBITS_ADD_ADVANCED_TEST() in the test CMakeLists.txt files.

Note that coverage results are reported to a separate dashboard. Developers requiring access to coverage results from outside of Sandia should send a note to the trilinos-framework mail list.

Valgrind Memory Testing: Most Secondary Tested Code [ Current Memory Testing Results]

!!! NOTE: REGULAR VALGRIND TESTING IS CURRENTLY DISABLED !!!

Every day a subset of Trilinos packages and tests are run through Valgrind and posed to CDash to find some forms of memory issues. Note that the memory checking tests can be quite long which is the reason why only a portion of memory issues are being checked currently.

Currently the test category used for Memory testing is Trilinos_TEST_CATEGORIES=NIGHTLY which matches to the CATEGORIES options BASIC or NIGHTLY passed into TRIBITS_ADD_TEST() and TRIBITS_ADD_ADVANCED_TEST() in the test CMakeLists.txt files.

Clang AddressSanitizer Memory Testing: All packages [ Current Memory Testing Results]

Every day a subset of Trilinos packages and tests are run through Clang AddressSanitizer and posed to CDash to find some forms of memory issues.

There are suppressions available at packages/framework/asan_assets/lsan.supp. This helps to suppress leak notifications for things like MPI over which Trilinos have no control. In addition, one needs to set LD_PRELOAD to contain the file packages/framework/asan_assets/dummy_dlclose.so (see https://github.com/google/sanitizers/issues/89).

The current way to reproduce Clang ASan errors is to use the correct GenConfig build name containing 'asan' (see the nightly dashboard). Once the build is complete, run the tests as follows:
export LSAN_OPTIONS=suppressions=${PATH_TO_TRILINOS}/packages/framework/asan_assets/lsan.supp
LD_PRELOAD=${PATH_TO_TRILINOS}/packages/framework/asan_assets/dummy_dlclose.so ctest ...
The reason for using this syntax is to ensure that LD_PRELOAD is only set as a part of the ctest process, since it is likely that setting this particular variable in this way is undesirable for other processes that you will run in the same shell. The LSAN_OPTIONS one is much safer, and can thus be set in a persistent manner. Also note that the supplied 'dummy_dlclose.so' was built for x86_64 architecture, and will likely not work on others (e.g. aarch64, ppc64le).

Scalability Testing: Secondary Tested Code

!!! NOT DEFINED YET !!! There is currently no support for automated scalability testing in the Trilinos CMake/CTest/CDash system.

Finding information on CDash

There is a wealth of information on CDash about the builds that are done, the version of Trilinos being tested, the commits that are pulled from the last time that build was run, etc. Below, are some tips on what of information there is and how to find and view it. As a frame of reference, we will use the CI build Linux-GCC-4.8.4-MPI_RELEASE_DEBUG_SHARED_PT_OPENMP_CI that runs on ceerws1113 which can be viewed here.

• Version of Trilinos being tested (and the prior version):
  • View Trilinos version natively in CDash:
    • Start on the cdash/index.php?project=Trilinos page view
    • Click on the build name (e.g. "Linux-GCC-4.8.4-MPI_RELEASE_DEBUG_SHARED_PT_OPENMP_CI")
    • On package-by-package build page cdash/index.php?project=Trilinos&parentid=<parent-build-id>, click on the number of updates in the "Update"|"Files" column for any package row displayed (e.g. for the "Teuchos" row).
    • On the package updates page cdash/viewUpdate.php?buildid=<pkg-build-id>, the current version and prior version SHA1s of Trilinos for this build are shown at the top of the page by the names "Revision" and "Prior Revision", respectively.
  • View notes files "TrilinosRepoVersion.txt" and "Updates.txt":
    • Start on the cdash/index.php?project=Trilinos page view
    • Click on the build name (e.g. "Linux-GCC-4.8.4-MPI_RELEASE_DEBUG_SHARED_PT_OPENMP_CI")
    • On package-by-package build page cdash/index.php?project=Trilinos&parentid=<parent-build-id>, click on any package name in the first "SubProject" column for any packages displayed (e.g. "Teuchos")
    • On the package build summary page cdash/buildSummary.php?buildid=<pkg-build-id>, click on the "view notes" link at the top ad the end of the "Build Name" field.
    • On the package notes page cdash/viewNotes.php?buildid=<pkg-build-id>:
      • Click on the "TrilinosRepoVersion.txt" file link to see the top commit from Trilinos being tested
      • Click on the "Updates.txt" file link to see the new commits pulled from the last time this build ran