TpetraCore_BlockCrsMatrix_MPI_4 failing in ATDM cuda builds #4257

fryeguy52 · 2019-01-24T17:53:13Z

CC: @trilinos/tpetra, @kddevin (Trilinos Data Services Product Lead), @bartlettroscoe, @fryeguy52

Next Action Status

With the merge of PR #4307 on to 'develop' on 2/4/2018, the test TpetraCore_BlockCrsMatrix_MPI_4 seems to be passing in all of the ATDM Trilinos builds on 2/5/2018. Next: Get PR #4326 merged which re-enables this test in the Trilinos CUDA PR build ...

Description

As shown in this query the test:

TpetraCore_BlockCrsMatrix_MPI_4

is failing in the builds:

Trilinos-atdm-waterman-cuda-9.2-debug
Trilinos-atdm-waterman-cuda-9.2-opt
Trilinos-atdm-waterman-cuda-9.2-release-debug
Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-debug
Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release
Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release-debug

It is failing with the following output:

p=0: *** Caught standard std::exception of type 'std::logic_error' :
 
  /home/jenkins/white/workspace/Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-debug/SRC_AND_BUILD/Trilinos/packages/tpetra/core/src/Tpetra_Experimental_BlockCrsMatrix_def.hpp:2825:
  
  Throw number = 1
  
  Throw test that evaluated to true: numBytesOut != numBytes
  
  unpackRow: numBytesOut = 4 != numBytes = 156.
 [FAILED]  (0.0877 sec) BlockCrsMatrix_double_int_int_Kokkos_Compat_KokkosCudaWrapperNode_write_UnitTest
 Location: /home/jenkins/white/workspace/Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-debug/SRC_AND_BUILD/Trilinos/packages/tpetra/core/test/Block/BlockCrsMatrix.cpp:859
 
[white23:102556] *** An error occurred in MPI_Allreduce
[white23:102556] *** reported by process [231079937,0]
[white23:102556] *** on communicator MPI_COMM_WORLD
[white23:102556] *** MPI_ERR_OTHER: known error not in list
[white23:102556] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[white23:102556] ***    and potentially your MPI job)

@kyungjoo-kim can you see if one of these commits may have caused this?

47f9cbe:  Tpetra - fix failing test
Author: Kyungjoo Kim (-EXP) <kyukim@bread.sandia.gov>
Date:   Tue Jan 22 11:24:43 2019 -0700

M	packages/tpetra/core/src/Tpetra_Experimental_BlockCrsMatrix_def.hpp

3e26a55:  Tpetra - fix warning error from mismatched virtual functions
Author: Kyungjoo Kim (-EXP) <kyukim@bread.sandia.gov>
Date:   Mon Jan 21 11:48:32 2019 -0700

M	packages/tpetra/core/src/Tpetra_Experimental_BlockCrsMatrix_decl.hpp
M	packages/tpetra/core/src/Tpetra_Experimental_BlockCrsMatrix_def.hpp

Current Status on CDash

The current status of these tests/builds for the current testing day can be found here

Steps to Reproduce

One should be able to reproduce this failure on ride or white as described in:

https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md

More specifically, the commands given for ride or white are provided at:

https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#ridewhite

The exact commands to reproduce this issue should be:

$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release
$ cmake \
 -GNinja \
 -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
 -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Tpetra=ON \
 $TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -q rhel7F -n 16 ctest -j16

The text was updated successfully, but these errors were encountered:

kyungjoo-kim · 2019-01-24T17:58:07Z

@fryeguy52 I will look at this problem today.

kyungjoo-kim · 2019-01-24T20:47:01Z

@fryeguy52 I followed the instruction to reproduce the error but I could not reproduce it on white. Could you double check if you can reproduce it ? I use

[kyukim @white11] master > git remote -v 
origin	https://github.com/trilinos/Trilinos.git (fetch)
origin	https://github.com/trilinos/Trilinos.git (push)
[kyukim @white11] master > git branch 
* develop
  master
[kyukim @white11] master > git log 
commit 01fb63caf88db53491f12afe0497c9d8f2cde09f
Merge: 4c4ecbf 3f3b1cc
Author: Mark Hoemmen <mhoemmen@users.noreply.github.com>
Date:   Mon Jan 21 13:12:15 2019 -0700

    Merge pull request #4224 from trilinos/Fix-4220
    
    MiniTensor: Attempt to fix #4220

This is output from white.

[kyukim @white11] atdm >  bsub -x -Is -q rhel7F -n 16 ctest -j16
***Forced exclusive execution
Job <43094> is submitted to queue <rhel7F>.
<<Waiting for dispatch ...>>
<<Starting on white22>>
Test project /ascldap/users/kyukim/Work/lib/trilinos/build/white/atdm
        Start   1: TpetraCore_Behavior_Default_MPI_4
        Start   2: TpetraCore_Behavior_Named_MPI_4
        Start   3: TpetraCore_Behavior_Off_MPI_4
        Start   4: TpetraCore_Behavior_On_MPI_4
  1/194 Test   #1: TpetraCore_Behavior_Default_MPI_4 ...........................................................   Passed    0.91 sec
        Start   5: TpetraCore_gemv_MPI_1
        Start   6: TpetraCore_gemm_m_eq_1_MPI_1
        Start   7: TpetraCore_gemm_m_eq_2_MPI_1
        Start   8: TpetraCore_gemm_m_eq_5_MPI_1
  2/194 Test   #2: TpetraCore_Behavior_Named_MPI_4 .............................................................   Passed    0.92 sec
        Start   9: TpetraCore_gemm_m_eq_13_MPI_1
        Start  11: TpetraCore_BlockMultiVector2_MPI_1
        Start  14: TpetraCore_BlockView_MPI_1
        Start  15: TpetraCore_BlockOps_MPI_1
  3/194 Test   #3: TpetraCore_Behavior_Off_MPI_4 ...............................................................   Passed    0.93 sec
        Start  10: TpetraCore_BlockMultiVector_MPI_4
  4/194 Test   #4: TpetraCore_Behavior_On_MPI_4 ................................................................   Passed    0.96 sec
        Start  12: TpetraCore_BlockCrsMatrix_MPI_4
  5/194 Test  #15: TpetraCore_BlockOps_MPI_1 ...................................................................   Passed    1.39 sec
        Start  16: TpetraCore_BlockExpNamespace_MPI_1
  6/194 Test  #14: TpetraCore_BlockView_MPI_1 ..................................................................   Passed    2.53 sec
        Start  31: TpetraCore_Core_initialize_where_tpetra_initializes_kokkos_MPI_1
  7/194 Test  #11: TpetraCore_BlockMultiVector2_MPI_1 ..........................................................   Passed    2.83 sec
        Start  32: TpetraCore_Core_ScopeGuard_where_tpetra_initializes_kokkos_MPI_1
  8/194 Test  #10: TpetraCore_BlockMultiVector_MPI_4 ...........................................................   Passed    2.82 sec
        Start  13: TpetraCore_BlockMap_MPI_4
  9/194 Test  #16: TpetraCore_BlockExpNamespace_MPI_1 ..........................................................   Passed    1.63 sec
        Start  33: TpetraCore_Core_initialize_where_user_initializes_kokkos_MPI_1
 10/194 Test  #31: TpetraCore_Core_initialize_where_tpetra_initializes_kokkos_MPI_1 ............................   Passed    0.81 sec
        Start  34: TpetraCore_Core_ScopeGuard_where_user_initializes_kokkos_MPI_1
 11/194 Test  #33: TpetraCore_Core_initialize_where_user_initializes_kokkos_MPI_1 ..............................   Passed    1.17 sec
        Start  39: TpetraCore_issue_434_already_initialized_MPI_1
 12/194 Test  #12: TpetraCore_BlockCrsMatrix_MPI_4 .............................................................   Passed    4.19 sec
...
100% tests passed, 0 tests failed out of 194

Subproject Time Summary:
Tpetra    = 1492.44 sec*proc (194 tests)

Total Test time (real) =  94.37 sec

fryeguy52 · 2019-01-24T21:45:40Z

@kyungjoo-kim Thanks for looking into this. I will try and reproduce and watch what it does tonight in testing.

bartlettroscoe · 2019-01-24T22:32:40Z

@kyungjoo-kim and @fryeguy52,

I just logged onto 'white' really quick and pulled Trilinos 'develop' as of version 3ef91e9 :

3ef91e9 "Merge Pull Request #4253 from trilinos/Trilinos/Fix-4234"
Author: trilinos-autotester <trilinos-autotester@trilinos.org>
Date:   Thu Jan 24 08:15:36 2019 -0700 (7 hours ago)

and following the instructions here I ran:

$ bsub -x -I -q rhel7F -n 16 \
  ./checkin-test-atdm.sh cuda-9.2-gnu-7.2.0-release-debug --enable-packages=TpetraCore --local-do-all

and it returned:

FAILED (NOT READY TO PUSH): Trilinos: white26

Thu Jan 24 15:06:51 MST 2019

Enabled Packages: TpetraCore

Build test results:
-------------------
0) MPI_RELEASE_DEBUG_SHARED_PT_OPENMP => Test case MPI_RELEASE_DEBUG_SHARED_PT_OPENMP was not run! => Does not affect push readiness! (-1.00 min)
1) cuda-9.2-gnu-7.2.0-release-debug => FAILED: passed=193,notpassed=1 => Not ready to push! (16.65 min)


REQUESTED ACTIONS: FAILED

The detailed test results showed:

$ grep -A 100 "failed out of" cuda-9.2-gnu-7.2.0-release-debug/ctest.out 
99% tests passed, 1 tests failed out of 194

Subproject Time Summary:
Tpetra    = 1094.70 sec*proc (194 tests)

Total Test time (real) = 139.04 sec

The following tests FAILED:
         12 - TpetraCore_BlockCrsMatrix_MPI_4 (Failed)
Errors while running CTest

kyungjoo-kim · 2019-01-24T23:09:03Z

I pull again and test it but I cannot reproduce the error with a commit 3ea64d1.

bartlettroscoe · 2019-01-24T23:13:52Z

@kyungjoo-kim said:

I pull again and test it but I cannot reproduce the error with a commit 3ea64d1.

Let's wait and see if @fryeguy52 can reproduce this on 'white' in his own account and go from there.

kddevin · 2019-01-29T23:31:15Z

Note that #4293 disabled this test; we'll need to re-enable it when this work is complete.
TpetraCore_BlockCrsMatrix_MPI_4_DISABLE

bartlettroscoe · 2019-01-29T23:47:50Z

@kddevin said:

Note that #4293 disabled this test; we'll need to re-enable it when this work is complete.
TpetraCore_BlockCrsMatrix_MPI_4_DISABLE

PR #4293 only disables that test for the CUDA PR build, not the ATDM Trilinos builds. (There is no relationship between these two sets of builds and that is on purpose.)

The question is if this failing test is something that should be fixed or not before that ATDM APPs get an updated version of Trilinos? Right now it is listed as ATDM Sev: Blocker. (But my guess is that EMPIRE is not getting impacted by this because we would have heard about.) Is this a real defect in Tpetra or just a problem with the test?

kyungjoo-kim · 2019-01-30T00:05:39Z

From the failed test message,

  unpackRow: numBytesOut = 4 != numBytes = 156.
 [FAILED]  (0.106 sec) BlockCrsMatrix_double_int_int_Kokkos_Compat_KokkosCudaWrapperNode_write_UnitTest
 Location: /home/jenkins/waterman/workspace/Trilinos-atdm-waterman-cuda-9.2-debug/SRC_AND_BUILD/Trilinos/packages/tpetra/core/test/Block/BlockCrsMatrix.cpp:859
 
[waterman3:115046] *** An error occurred in MPI_Allreduce
[waterman3:115046] *** reported by process [3797417985,0]
[waterman3:115046] *** on communicator MPI_COMM_WORLD
[waterman3:115046] *** MPI_ERR_OTHER: known error not in list
[waterman3:115046] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[waterman3:115046] ***    and potentially your MPI job)

I first examine the packing and unpacking routine if there is some mistake in dualview sync as the input array of the unpacking, imports, is all zeros. Then, I see the All reduce error. I am not sure which trigger which error. However, it is also possible that MPI_Allreduce gets an error and it causes corruption in the importer. The other way is also possible. Something is not synced from device and it causes the MPI error. From the other test in BlockCrs, creating a map (this is not related to blockcrs code but it just happens in the blockcrs code) also fails with mpi all reduce.

kyungjoo-kim · 2019-01-31T19:58:49Z

@bartlettroscoe PR #4307 will fix the block crs unit test failures. After the PR is merged, would you please re-enable the test ?

bartlettroscoe · 2019-01-31T20:14:35Z

@kyungjoo-kim said:

@bartlettroscoe PR #4307 will fix the block crs unit test failures. After the PR is merged, would you please re-enable the test ?

Thanks for fix!

Someone will need to revert PR #4293 after we confirm that these tests are fixed in the ATDM build (where this tests was never disabled, PR #4293 only disabled them in the Trilinos PR build controlled by the @trilinos/framework team).

mhoemmen · 2019-02-05T01:22:27Z

@kyungjoo-kim Can we reenable that test now?

bartlettroscoe · 2019-02-05T16:27:25Z

With the merge of PR #4307 on to 'develop' on 2/4/2018, the test TpetraCore_BlockCrsMatrix_MPI_4 seems to be passing in all of the ATDM Trilinos builds on 2/5/2018. See table below.

I will leave this open until the CUDA PR testing gets this test enabled again by reverting PR #4293.

Tests with issue trackers Passed: twip=6 (Testing day 2019-02-05)

Site	Build Name	Test Name	Status	Details	Consecutive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
waterman	Trilinos-atdm-waterman-cuda-9.2-debug	TpetraCore_BlockCrsMatrix_MPI_4	Passed	Completed	1	12	17	#4257
waterman	Trilinos-atdm-waterman-cuda-9.2-opt	TpetraCore_BlockCrsMatrix_MPI_4	Passed	Completed	1	12	17	#4257
waterman	Trilinos-atdm-waterman-cuda-9.2-release-debug	TpetraCore_BlockCrsMatrix_MPI_4	Passed	Completed	1	12	15	#4257
white	Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-debug	TpetraCore_BlockCrsMatrix_MPI_4	Passed	Completed	1	12	20	#4257
white	Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release-debug	TpetraCore_BlockCrsMatrix_MPI_4	Passed	Completed	1	10	17	#4257
white	Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-release	TpetraCore_BlockCrsMatrix_MPI_4	Passed	Completed	1	11	18	#4257

bartlettroscoe · 2019-02-05T16:34:32Z

FYI: I created the revert PR #4326 to re-enable this test in the Trilinos CUDA PR build. Just need someone to approve this PR and get it merged. Then we can close this issue.

mhoemmen · 2019-02-05T16:37:28Z

Thanks Ross! I just approved the PR.

fryeguy52 added this to the Keep promoted "ATDM" builds of Trilinos clean milestone Jan 24, 2019

kyungjoo-kim self-assigned this Jan 24, 2019

kyungjoo-kim mentioned this issue Jan 31, 2019

Tpetra: Fix BlockCrs unit test on CUDA. #4307

Merged

bartlettroscoe mentioned this issue Feb 5, 2019

Revert "Disable Tpetra test for CUDA PR build" #4326

Merged

kyungjoo-kim closed this as completed Feb 5, 2019

mhoemmen mentioned this issue Feb 6, 2019

Tpetra::DistObject::doTransferNew: Don't recreate DualViews for LID arrays each call #4328

Merged

trilinos-autotester mentioned this issue Dec 14, 2021

MueLu: Make Phase2a do the same as ML #9747

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TpetraCore_BlockCrsMatrix_MPI_4 failing in ATDM cuda builds #4257

TpetraCore_BlockCrsMatrix_MPI_4 failing in ATDM cuda builds #4257

fryeguy52 commented Jan 24, 2019 •

edited by bartlettroscoe

Loading

kyungjoo-kim commented Jan 24, 2019

kyungjoo-kim commented Jan 24, 2019

fryeguy52 commented Jan 24, 2019

bartlettroscoe commented Jan 24, 2019

kyungjoo-kim commented Jan 24, 2019

bartlettroscoe commented Jan 24, 2019

kddevin commented Jan 29, 2019

bartlettroscoe commented Jan 29, 2019 •

edited

Loading

kyungjoo-kim commented Jan 30, 2019

kyungjoo-kim commented Jan 31, 2019

bartlettroscoe commented Jan 31, 2019

mhoemmen commented Feb 5, 2019

bartlettroscoe commented Feb 5, 2019

bartlettroscoe commented Feb 5, 2019

mhoemmen commented Feb 5, 2019

TpetraCore_BlockCrsMatrix_MPI_4 failing in ATDM cuda builds #4257

TpetraCore_BlockCrsMatrix_MPI_4 failing in ATDM cuda builds #4257

Comments

fryeguy52 commented Jan 24, 2019 • edited by bartlettroscoe Loading

Next Action Status

Description

Current Status on CDash

Steps to Reproduce

kyungjoo-kim commented Jan 24, 2019

kyungjoo-kim commented Jan 24, 2019

fryeguy52 commented Jan 24, 2019

bartlettroscoe commented Jan 24, 2019

kyungjoo-kim commented Jan 24, 2019

bartlettroscoe commented Jan 24, 2019

kddevin commented Jan 29, 2019

bartlettroscoe commented Jan 29, 2019 • edited Loading

kyungjoo-kim commented Jan 30, 2019

kyungjoo-kim commented Jan 31, 2019

bartlettroscoe commented Jan 31, 2019

mhoemmen commented Feb 5, 2019

bartlettroscoe commented Feb 5, 2019

Tests with issue trackers Passed: twip=6 (Testing day 2019-02-05)

bartlettroscoe commented Feb 5, 2019

mhoemmen commented Feb 5, 2019

fryeguy52 commented Jan 24, 2019 •

edited by bartlettroscoe

Loading

bartlettroscoe commented Jan 29, 2019 •

edited

Loading