Xpetra_MultiVector_UnitTests_MPI_4 failing in Trilinos builds starting 2021-10-02

[ZUUL42]

CC: @trilinos/xpetra, @jhux2 (Trilinos: Data Services, Triage Contact: Jonathan Hu)

Next Action Status

Description

As shown in this query (click "Shown Matching Output" in upper right) the tests:

• Xpetra_MultiVector_UnitTests_MPI_4

in the builds:

• Trilinos-atdm-ats1-hsw_intel-19.0.4_mpich-7.7.15_openmp_static_opt
• Trilinos-atdm-ats2-cuda-10.1.243-gnu-7.3.1-spmpi-rolling_complex_static_opt
• Trilinos-atdm-ats2-cuda-10.1.243-gnu-7.3.1-spmpi-rolling_complex_static_opt_cuda-aware-mpi
• Trilinos-atdm-ats2-cuda-10.1.243-gnu-7.3.1-spmpi-rolling_static_opt
• Trilinos-atdm-ats2-cuda-10.1.243-gnu-7.3.1-spmpi-rolling_static_opt_cuda-aware-mpi
• Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_serial_static_opt
• Trilinos-atdm-cee-rhel7_clang-9.0.1_openmpi-4.0.3_serial_static_opt
• Trilinos-atdm-cee-rhel7_cuda-10.1.243_gnu-7.2.0_openmpi-4.0.3_shared_opt
• Trilinos-atdm-cee-rhel7_gnu-7.2.0_openmpi-4.0.3_serial_shared_opt
• Trilinos-atdm-cee-rhel7_intel-19.0.3_intelmpi-2018.4_serial_static_opt
• Trilinos-atdm-cee-rhel7_intel-19.0.3_mpich2-3.2_openmp_static_opt
• Trilinos-atdm-cts1-intel-19.0.4_openmpi-4.0.3_openmp_static_opt
• Trilinos-atdm-cts1empire-intel-18.0.2_openmpi-4.0.1_openmp_static_opt
• Trilinos-atdm-tlcc2-intel-opt-openmp
• Trilinos-atdm-van1-tx2_arm-20.1_openmpi-4.0.5_openmp_static_opt

started failing on testing day 2021-10-02.

 Test that code {m1n2.update(rnd,m2n2,rnd);} throws std::runtime_error: failed (code did not throw an exception at all)
 Test that code {m1n2.update(rnd,m2n2 ,rnd,m1n2_2,rnd);} throws std::runtime_error: failed (code did not throw an exception at all)
 Test that code {m1n2.update(rnd,m2n2 ,rnd,m1n2_2,rnd);} throws std::runtime_error: failed (code did not throw an exception at all)
 Test that code {m1n2.update(rnd,m1n2_2,rnd,m2n2 ,rnd);} throws std::runtime_error: failed (code did not throw an exception at all)
 Test that code {m1n2.update(rnd,m1n2_2,rnd,m2n2 ,rnd);} throws std::runtime_error: failed (code did not throw an exception at all)
 Test that code {m1n2.update(rnd,m2n2 ,rnd,m2n2 ,rnd);} throws std::runtime_error: failed (code did not throw an exception at all)
 Test that code {m1n2.update(rnd,m2n2 ,rnd,m2n2 ,rnd);} throws std::runtime_error: failed (code did not throw an exception at all)

Current Status on CDash

Run the above query adjusting the "Begin" and "End" dates to match today any other date range or just click "CURRENT" in the top bar to see results for the current testing day.

Steps to Reproduce

One should be able to reproduce this failure as described in:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md

and the system-specific instructions at:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#specific-instructions-for-each-system

Just log into any of the associated machines and copy and paste the full CDash build name <build-name> listed above and run commands like:

$ cd <some_build_dir>/

$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh <build-name>

$ cmake \
 -GNinja \
 -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
 -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_<package-name>=ON \
 $TRILINOS_DIR

$ make NP=16

$ <command-to-run-on-compute-node> ctest -j4

where <package-name> is any package that you want to enable to reproduce build and/or test results.

Again, for exact system-specific details on what commands to run to build and run tests, see:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#specific-instructions-for-each-system

If you can't figure out what commands to run to reproduce the problem given this documentation, then please post a comment here and we will give you the exact minimal commands.

[cgcgcg]

This should be fixed since PR #10186 merged. I don't see any failures for last night, but we can give it a couple of days before closing.

[ZUUL42]

I show the last time this error presented itself was Feb 22, the same day but before the PR merged.
testing.sandia.gov/cdash
Nothing since.