Reduce all CUDA builds to use ctest -j4 to avoid flaky UVM out of memory failures

[bartlettroscoe]

As described in kokkos/kokkos#2330 tests on GPUs may randomly fail with errors like:

terminate called after throwing an instance of 'std::runtime_error'
  what():  cudaDeviceSynchronize() error( cudaErrorIllegalAddress): an illegal memory access was encountered <file-name>:<line-number>

This may be caused by the memory on the GPUs being overloaded when running tests.

It was recommended by the Kokkos developers to use a smaller parallel testing level.

I think is is appropriate to use ctest -j4 since that will allow just one 4-process test to run at a time but will allow four 1-process tests (or two 2-process tests) to run at a time.

[bartlettroscoe]

FYI: As shown in this query, the test:

• MueLu_DriverTpetraSingleReduceCG_MPI_4

is randomly failing in the build:

• Trilinos-atdm-waterman_cuda-9.2_shared_opt

showing the error:

Laplace2D: MxM: A x P
Laplace2D: MxM: P' x (AP) (implicit)
terminate called after throwing an instance of 'std::runtime_error'
  what():  cudaDeviceSynchronize() error( cudaErrorIllegalAddress): an illegal memory access was encountered /home/atdm-devops-admin/jenkins/waterman/Trilinos-atdm-waterman_cuda-9.2_shared_opt/SRC_AND_BUILD/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Instance.cpp:120
Traceback functionality not available

[waterman3:24936] *** Process received signal ***
[waterman3:24936] Signal: Aborted (6)
[waterman3:24936] Signal code:  (-6)

Therefore, this could be due to overloading memory in this build.

Need to reduce parallel level on all builds to ctest -j4 before I create new ATDM Trilinos GitHub issues for failures like this.

[bartlettroscoe]

FYI: All of the ATDM Trilinos test failures showing errors like:

terminate called after throwing an instance of 'std::runtime_error'
  what():  cudaDeviceSynchronize() error( cudaErrorIllegalAddress): an illegal memory access was encountered /home/atdm-devops-admin/jenkins/waterman/Trilinos-atdm-waterman_cuda-9.2_shared_opt/SRC_AND_BUILD/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Instance.cpp:120
Traceback functionality not available

[waterman7:46691] *** Process received signal ***
[waterman7:46691] Signal: Aborted (6)
[waterman7:46691] Signal code:  (-6)

in the last month are shown in this query for 2019-09-08 through 2019-10-08 which only impacts several 4-proc MueLu tests in the following 'waterman' builds:

• Trilinos-atdm-waterman-cuda-9.2-debug
• Trilinos-atdm-waterman-cuda-9.2-release-debug
• Trilinos-atdm-waterman_cuda-9.2_shared_opt

and the single STK test STKUnit_tests_stk_ngp_test_utest_MPI_4 and only in CUDA builds on 'ride':

• Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release-debug-pt

(This STK test failure was reported long ago in #4551.)

If these CUDA errors are really being caused by running out of GPU memory, then why are we not seeing large Panzer tests failing as well? Is it more likely that these errors are caused by UVM memory usage problems in MueLu and STK?

In any case, I will reduce the parallel test level for these CUDA builds from ctest -j8 to ctest -j4 and see what happens. NOTE: Since all of the failing MueLu tests shown in the above query are 4-proc MPI tests, running with ctest -j4 means that only one of thse 4-proc tests will be running at a time. Therefore, w.r.t. these tests, this should be equivalent to running ctest -j1.

[bartlettroscoe]

PR is #6069

[bartlettroscoe]

FYI: I manually merged the topic branch in PR #6069 to 'atdm-nightly' in commit 2317c4d. Therefore, this will be running in ATDM Trilinos builds starting tomorrow.

[bartlettroscoe]

FYI: Looks like there have been some tests failing showing errors (here) like:

terminate called after throwing an instance of 'std::runtime_error'
  what():  cudaDeviceSynchronize() error( cudaErrorMemoryAllocation): out of memory /home/atdm-devops-admin/jenkins/waterman/Trilinos-atdm-waterman_cuda-9.2_shared_opt/SRC_AND_BUILD/Trilinos/packages/kokkos/core/src/Cuda/Kokkos_Cuda_Instance.cpp:120

But looking across all ATDM Trilinos builds between 9/1/2019 and 10/10/2019 that showed this error in this query there were only two such failing tests:

Build Name	Test Name	Status	Time	Details	Build Time	Processors
Trilinos-atdm-waterman-_cuda-9.2_shared_opt	MueLu_Aggregation_MPI_4	Failed	6s 980ms	Completed (Failed)	2019-09-12T03:06:13 MDT	4
Trilinos-atdm-waterman_cuda-9.2_shared_opt	MueLu_Driver_TogglePFactory-_semi_tent_line_Tpetra_MPI_4	Failed	29s 230ms	Completed (Failed)	2019-09-19T03:06:57 MDT	4

Therefore, this error is not common at all. This move to ctest -j4 should eliminate these since these are 4-proc MPI tests and each of these will run by themselves after this change (which is active today).

[bartlettroscoe]

Looking at the wall-clock times for some of the CUDA builds like for:

• Trilinos-atdm-waterman-cuda-9.2-opt
• Trilinos-atdm-waterman-cuda-9.2-debug
• Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-debug

the wall-clock times for running the tests for these builds did not go up by all that much from before 2019-11-22 to after 2019-11-22 (when PR #6632 was merged). From the naked eye it looks like it might have gone up 10% or 20% but not that much. We would have to to compute the means but that is hard to do because new tests are being added and expended so it is hard to make a fair comparison.

Therefore, my initial observation is that turning down the parallel testing level to ctest -j4 looks like it did not slow down the test suite by that much for most builds. Therefore, if this helps to improve the robustness of the test suite when running on the GPU, then is a win win situation.

Closing as complete.

[bartlettroscoe]

Closing as complete for real :-)