Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tandem on GPUs #50

Closed
Thomas-Ulrich opened this issue Aug 25, 2023 · 14 comments
Closed

tandem on GPUs #50

Thomas-Ulrich opened this issue Aug 25, 2023 · 14 comments

Comments

@Thomas-Ulrich
Copy link
Collaborator

I've installed a cuda-aware petsc and associated tandem with:
spack install -j 50 tandem@main polynomial_degree=2 domain_dimension=3 ^petsc +cuda ^cuda@11 ^openmpi+cuda

This yields the following dependencies:

(base) ulrich@heisenbug:/import/exception-dump/ulrich/spack/var/spack/repos/builtin/packages/tandem$ spack spec tandem@main polynomial_degree=2 domain_dimension=3 ^petsc +cuda ^cuda@11 ^openmpi+cuda
Input spec
--------------------------------
 -   tandem@main domain_dimension=3 polynomial_degree=2
 -       ^cuda@11
 -       ^openmpi+cuda
 -       ^petsc+cuda

Concretized
--------------------------------
[+]  tandem@main%gcc@11.2.0~ipo~libxsmm build_system=cmake build_type=Release domain_dimension=3 generator=make min_quadrature_order=0 polynomial_degree=2 arch=linux-debian11-zen2
[+]      ^cmake@3.26.3%gcc@11.2.0~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=linux-debian11-zen2
[+]          ^curl@8.1.2%gcc@11.2.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-debian11-zen2
[+]              ^nghttp2@1.52.0%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]              ^openssl@3.1.1%gcc@11.2.0~docs~shared build_system=generic certs=mozilla arch=linux-debian11-zen2
[+]                  ^ca-certificates-mozilla@2023-05-30%gcc@11.2.0 build_system=generic arch=linux-debian11-zen2
[+]          ^ncurses@6.4%gcc@11.2.0~symlinks+termlib abi=none build_system=autotools arch=linux-debian11-zen2
[+]      ^eigen@3.4.0%gcc@11.2.0~ipo build_system=cmake build_type=RelWithDebInfo generator=make arch=linux-debian11-zen2
[+]      ^gmake@4.4.1%gcc@11.2.0~guile build_system=autotools arch=linux-debian11-zen2
[+]      ^lua@5.4.4%gcc@11.2.0~pcfile+shared build_system=makefile fetcher=curl arch=linux-debian11-zen2
[+]          ^readline@8.2%gcc@11.2.0 build_system=autotools patches=bbf97f1 arch=linux-debian11-zen2
[+]          ^unzip@6.0%gcc@11.2.0 build_system=makefile arch=linux-debian11-zen2
[+]      ^metis@5.1.0%gcc@11.2.0~gdb+int64~ipo~real64+shared build_system=cmake build_type=Release generator=make patches=4991da9,93a7903,b1225da arch=linux-debian11-zen2
[+]      ^openmpi@4.1.5%gcc@11.2.0~atomics+cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix+romio+rsh~singularity+static+vt+wrapper-rpath build_system=autotools cuda_arch=none fabrics=none schedulers=none arch=linux-debian11-zen2
[+]          ^cuda@11.8.0%gcc@11.2.0~allow-unsupported-compilers~dev build_system=generic arch=linux-debian11-zen2
[+]              ^libxml2@2.10.3%gcc@11.2.0~python build_system=autotools arch=linux-debian11-zen2
[+]                  ^xz@5.4.1%gcc@11.2.0~pic build_system=autotools libs=shared,static arch=linux-debian11-zen2
[+]          ^hwloc@2.9.1%gcc@11.2.0~cairo+cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools cuda_arch=none libs=shared,static arch=linux-debian11-zen2
[+]              ^libpciaccess@0.17%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]                  ^util-macros@1.19.3%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]          ^numactl@2.0.14%gcc@11.2.0 build_system=autotools patches=4e1d78c,62fc8a8,ff37630 arch=linux-debian11-zen2
[+]              ^autoconf@2.69%gcc@11.2.0 build_system=autotools patches=35c4492,7793209,a49dd5b arch=linux-debian11-zen2
[+]              ^automake@1.16.5%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]              ^libtool@2.4.7%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]              ^m4@1.4.19%gcc@11.2.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-debian11-zen2
[+]                  ^libsigsegv@2.14%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]          ^openssh@9.3p1%gcc@11.2.0+gssapi build_system=autotools arch=linux-debian11-zen2
[+]              ^krb5@1.20.1%gcc@11.2.0+shared build_system=autotools arch=linux-debian11-zen2
[+]                  ^bison@3.8.2%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]                  ^findutils@4.9.0%gcc@11.2.0 build_system=autotools patches=440b954 arch=linux-debian11-zen2
[+]                  ^gettext@0.21.1%gcc@11.2.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=linux-debian11-zen2
[+]                      ^tar@1.34%gcc@11.2.0 build_system=autotools zip=pigz arch=linux-debian11-zen2
[+]                          ^pigz@2.7%gcc@11.2.0 build_system=makefile arch=linux-debian11-zen2
[+]                          ^zstd@1.5.5%gcc@11.2.0+programs build_system=makefile compression=none libs=shared,static arch=linux-debian11-zen2
[+]              ^libedit@3.1-20210216%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]              ^libxcrypt@4.4.35%gcc@11.2.0~obsolete_api build_system=autotools patches=4885da3 arch=linux-debian11-zen2
[+]          ^perl@5.38.0%gcc@11.2.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-debian11-zen2
[+]              ^berkeley-db@18.1.40%gcc@11.2.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-debian11-zen2
[+]              ^bzip2@1.0.8%gcc@11.2.0~debug~pic+shared build_system=generic arch=linux-debian11-zen2
[+]              ^gdbm@1.23%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]          ^pkgconf@1.9.5%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]          ^pmix@4.2.4%gcc@11.2.0~docs+pmi_backwards_compatibility~python~restful build_system=autotools arch=linux-debian11-zen2
[+]              ^libevent@2.1.12%gcc@11.2.0+openssl build_system=autotools arch=linux-debian11-zen2
[+]      ^parmetis@4.0.3%gcc@11.2.0~gdb+int64~ipo+shared build_system=cmake build_type=Release generator=make patches=4f89253,50ed208,704b84f arch=linux-debian11-zen2
[+]      ^petsc@3.18.5%gcc@11.2.0~X~batch~cgns~complex+cuda~debug+double~exodusii~fftw+fortran~giflib+hdf5~hpddm~hwloc+hypre+int64~jpeg~knl~kokkos~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi+mumps~openmp~p4est~parmmg~ptscotch~random123~rocm~saws+scalapack+shared~strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind build_system=generic clanguage=C cuda_arch=none memalign=32 arch=linux-debian11-zen2
[+]          ^diffutils@3.9%gcc@11.2.0 build_system=autotools arch=linux-debian11-zen2
[+]              ^libiconv@1.17%gcc@11.2.0 build_system=autotools libs=shared,static arch=linux-debian11-zen2
[+]          ^hdf5@1.14.2%gcc@11.2.0~cxx~fortran~hl~ipo~java~map+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make arch=linux-debian11-zen2
[+]          ^hypre@2.28.0%gcc@11.2.0~caliper~complex~cuda~debug+fortran~gptune+int64~internal-superlu~mixedint+mpi~openmp~rocm+shared~superlu-dist~sycl~umpire~unified-memory build_system=autotools arch=linux-debian11-zen2
[+]          ^mumps@5.5.1%gcc@11.2.0~blr_mt+complex+double+float~incfort~int64+metis+mpi~openmp+parmetis~ptscotch~scotch+shared build_system=generic patches=373d736 arch=linux-debian11-zen2
[+]          ^netlib-scalapack@2.2.0%gcc@11.2.0~ipo~pic+shared build_system=cmake build_type=Release generator=make patches=072b006,1c9ce5f,244a9aa arch=linux-debian11-zen2
[+]          ^openblas@0.3.23%gcc@11.2.0~bignuma~consistent_fpcsr+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-debian11-zen2
[e]          ^python@3.9.2%gcc@11.2.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma+nis~optimizations+pic+pyexpat~pythoncmd+readline+shared+sqlite3+ssl+tix+tkinter+uuid+zlib build_system=generic patches=0d98e93,f2fd060 arch=linux-debian11-zen2
[+]          ^superlu-dist@8.1.2%gcc@11.2.0~cuda+int64~ipo~openmp~rocm+shared build_system=cmake build_type=Release generator=make arch=linux-debian11-zen2
[+]      ^zlib@1.2.13%gcc@11.2.0+optimize+pic+shared build_system=makefile arch=linux-debian11-zen2

I've tested it on the Ridgecrest example from the paper (note that I am able to run it full CPUs).
One node (note the added Petsc arguments -mg_levels_mat_type aijcusparse -vec_type cuda -mat_type aijcusparse):

(base) ulrich@heisenbug:/export/dump/ulrich/section_8_2/scenario-rc$ static ridge.toml  --output ridgecrest  --mg_strategy twolevel --mg_coarse_level 1  --petsc -ksp_view -ksp_monitor -ksp_converged_reason -ksp_max_it 400 -pc_type mg -mg_levels_ksp_max_it 4 -mg_levels_ksp_type cg -mg_levels_pc_type bjacobi -options_left -ksp_rtol 1.0e-6 -mg_coarse_pc_type gamg -mg_coarse_ksp_type cg -mg_coarse_ksp_rtol 1.0e-1 -mg_coarse_ksp_converged_reason -ksp_type gcr -mg_levels_mat_type aijcusparse -vec_type cuda -mat_type aijcusparse

               ___          ___         _____         ___          ___
      ___     /  /\        /__/\       /  /::\       /  /\        /__/\
     /  /\   /  /::\       \  \:\     /  /:/\:\     /  /:/_      |  |::\
    /  /:/  /  /:/\:\       \  \:\   /  /:/  \:\   /  /:/ /\     |  |:|:\
   /  /:/  /  /:/~/::\  _____\__\:\ /__/:/ \__\:| /  /:/ /:/_  __|__|:|\:\
  /  /::\ /__/:/ /:/\:\/__/::::::::\\  \:\ /  /://__/:/ /:/ /\/__/::::| \:\
 /__/:/\:\\  \:\/:/__\/\  \:\~~\~~\/ \  \:\  /:/ \  \:\/:/ /:/\  \:\~~\__\/
 \__\/  \:\\  \::/      \  \:\  ~~~   \  \:\/:/   \  \::/ /:/  \  \:\
      \  \:\\  \:\       \  \:\        \  \::/     \  \:\/:/    \  \:\
       \__\/ \  \:\       \  \:\        \__\/       \  \::/      \  \:\
              \__\/        \__\/                     \__\/        \__\/

                          tandem version 9e964fe

                       stack size limit = unlimited

                              Worker affinity
    0123456789|0123456789|0123456789|0123456789|0123456789|0123456789|
    0123456789|0123456789|0123456789|0123456789|0123456789|0123456789|
    0123456789|0123456789|0123456789|0123456789|0123456789|0123456789|
    0123456789|0123456789|0123456789|0123456789|0123456789|0123456789|
    0123456789|012345


Ridge:create()
parsing fault_many_trace.csv
DOFs: 12634590
Mesh size: 36.7959
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and https://petsc.org/release/faq/
[0]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash.

Same error on 2 nodes:

(base) ulrich@heisenbug:/export/dump/ulrich/section_8_2/scenario-rc$ sh run2.sh 

               ___          ___         _____         ___          ___
      ___     /  /\        /__/\       /  /::\       /  /\        /__/\
     /  /\   /  /::\       \  \:\     /  /:/\:\     /  /:/_      |  |::\
    /  /:/  /  /:/\:\       \  \:\   /  /:/  \:\   /  /:/ /\     |  |:|:\
   /  /:/  /  /:/~/::\  _____\__\:\ /__/:/ \__\:| /  /:/ /:/_  __|__|:|\:\
  /  /::\ /__/:/ /:/\:\/__/::::::::\\  \:\ /  /://__/:/ /:/ /\/__/::::| \:\
 /__/:/\:\\  \:\/:/__\/\  \:\~~\~~\/ \  \:\  /:/ \  \:\/:/ /:/\  \:\~~\__\/
 \__\/  \:\\  \::/      \  \:\  ~~~   \  \:\/:/   \  \::/ /:/  \  \:\
      \  \:\\  \:\       \  \:\        \  \::/     \  \:\/:/    \  \:\
       \__\/ \  \:\       \  \:\        \__\/       \  \::/      \  \:\
              \__\/        \__\/                     \__\/        \__\/

                          tandem version 9e964fe

                       stack size limit = unlimited

                              Worker affinity
    0---------|----------|----------|----------|----------|----------|
    ----------|----------|----------|----------|----------|----------|
    --------8-|----------|----------|----------|----------|----------|
    ----------|----------|----------|----------|----------|----------|
    ----------|------


Ridge:create()
parsing fault_many_trace.csv
Ridge:create()
parsing fault_many_trace.csv
DOFs: 12634590
Mesh size: 36.7959
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and https://petsc.org/release/faq/
[0]PETSC ERROR: or try https://docs.nvidia.com/cuda/cuda-memcheck/index.html on NVIDIA CUDA systems to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash.
```
launch script:
```
(base) ulrich@heisenbug:/export/dump/ulrich/section_8_2/scenario-rc$ cat run2.sh 
#!/bin/bash
mpiexec -n 2  static ridge.toml  --output ridgecrest  --mg_strategy twolevel --mg_coarse_level 1  --petsc -ksp_view -ksp_monitor -ksp_converged_reason -ksp_max_it 400 -pc_type mg -mg_levels_ksp_max_it 4 -mg_levels_ksp_type cg -mg_levels_pc_type bjacobi -options_left -ksp_rtol 1.0e-6 -mg_coarse_pc_type gamg -mg_coarse_ksp_type cg -mg_coarse_ksp_rtol 1.0e-1 -mg_coarse_ksp_converged_reason -ksp_type gcr -mg_levels_mat_type aijcusparse -vec_type cuda -mat_type aijcusparse
```
@hpc4geo
Copy link
Collaborator

hpc4geo commented Aug 25, 2023 •
edited
Loading

Okay. Couple of things.

[1] Lets first establish that the PETSc + GPU build is working with a very simple example. Let's start by not using multigrid at all. Please try this and upload what is sent to stdout.

mpiexec -n 2 static ridge.toml --output ridgecrest --matrix_free false --petsc -ksp_max_it 400 -pc_type jacobi -ksp_rtol 1.0e-6 -ksp_type gcr -vec_type cuda -mat_type aijcusparse -ksp_view -ksp_monitor -ksp_converged_reason -options_left -log_view

[2] In general, debugging this will be impossible with an optimized build of PETSc. For testing you absolutely must use a debug build of PETSc. Does spack support this?

@Thomas-Ulrich
Copy link
Collaborator Author

I get the following error right at the start:

(base) ulrich@heisenbug:/export/dump/ulrich/section_8_2/scenario-rc$ mpiexec -n 2 static ridge.toml --output ridgecrest --matrix_free false --petsc -ksp_max_it 400 -pc_type jacobi -ksp_rtol 1.0e-6 -ksp_type gcr -vec_type cuda -mat_type aijcusparse -ksp_view -ksp_monitor -ksp_converged_reason -options_left -log_view
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Petsc has generated inconsistent data
[1]PETSC ERROR: Invalid stack size 0, pop register_PCs /tmp/ulrich/spack-stage/spack-stage-tandem-main-wnfinsgrh3swjkyofitwrffzldwourlj/spack-src/app/pc/register.cpp:18.

[1]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[1]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[1]PETSC ERROR: Option left: name:-ksp_max_it value: 400
[1]PETSC ERROR: Option left: name:-ksp_monitor (no value)
[1]PETSC ERROR: Option left: name:-ksp_rtol value: 1.0e-6
[1]PETSC ERROR: Option left: name:-ksp_type value: gcr
[1]PETSC ERROR: Option left: name:-ksp_view (no value)
[1]PETSC ERROR: Option left: name:-mat_type value: aijcusparse
[1]PETSC ERROR: Option left: name:-options_left (no value)
[1]PETSC ERROR: Option left: name:-pc_type value: jacobi
[1]PETSC ERROR: Option left: name:-vec_type value: cuda
[1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.18.5, Feb 27, 2023 
[1]PETSC ERROR: --petsc on a  named heisenbug by ulrich Thu Aug 31 18:31:30 2023
[1]PETSC ERROR: Configure options --prefix=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 --with-make-exec=make --with-cc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpicc --with-cxx=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpic++ --with-fc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=1 --with-openmp=0 --with-64-bit-indices=1 --with-blaslapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openblas-0.3.23-kxy7bxom4lam3wx44st22oytplyuisby/lib/libopenblas.so --with-memalign=32 --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u --with-hip=0 --with-metis=1 --with-metis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/include --with-metis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/lib/libmetis.so --with-hypre=1 --with-hypre-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/include --with-hypre-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/include --with-parmetis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/lib/libparmetis.so --with-kokkos=0 --with-kokkos-kernels=0 --with-superlu_dist=1 --with-superlu_dist-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/include --with-superlu_dist-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/include --with-hdf5-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/lib/libhdf5.so --with-zlib=1 --with-zlib-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/include --with-zlib-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/lib/libz.so --with-mumps=1 --with-mumps-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/include --with-mumps-lib="/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libsmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libzmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libcmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libdmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libmumps_common.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libpord.so" --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/netlib-scalapack-2.2.0-vs6d5fhfa4i77gfo5iurpifbgxwfrmbp/lib/libscalapack.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0
[1]PETSC ERROR: #1 register_PCs() at /tmp/ulrich/spack-stage/spack-stage-tandem-main-wnfinsgrh3swjkyofitwrffzldwourlj/spack-src/app/pc/register.cpp:18
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Petsc has generated inconsistent data
[0]PETSC ERROR: Invalid stack size 0, pop register_PCs /tmp/ulrich/spack-stage/spack-stage-tandem-main-wnfinsgrh3swjkyofitwrffzldwourlj/spack-src/app/pc/register.cpp:18.

[0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[0]PETSC ERROR: Option left: name:-ksp_max_it value: 400
[0]PETSC ERROR: Option left: name:-ksp_monitor (no value)
[0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.0e-6
[0]PETSC ERROR: Option left: name:-ksp_type value: gcr
[0]PETSC ERROR: Option left: name:-ksp_view (no value)
[0]PETSC ERROR: Option left: name:-mat_type value: aijcusparse
[0]PETSC ERROR: Option left: name:-options_left (no value)
[0]PETSC ERROR: Option left: name:-pc_type value: jacobi
[0]PETSC ERROR: Option left: name:-vec_type value: cuda
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.18.5, Feb 27, 2023 
[0]PETSC ERROR: --petsc on a  named heisenbug by ulrich Thu Aug 31 18:31:30 2023
[0]PETSC ERROR: Configure options --prefix=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 --with-make-exec=make --with-cc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpicc --with-cxx=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpic++ --with-fc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=1 --with-openmp=0 --with-64-bit-indices=1 --with-blaslapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openblas-0.3.23-kxy7bxom4lam3wx44st22oytplyuisby/lib/libopenblas.so --with-memalign=32 --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u --with-hip=0 --with-metis=1 --with-metis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/include --with-metis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/lib/libmetis.so --with-hypre=1 --with-hypre-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/include --with-hypre-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/include --with-parmetis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/lib/libparmetis.so --with-kokkos=0 --with-kokkos-kernels=0 --with-superlu_dist=1 --with-superlu_dist-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/include --with-superlu_dist-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/include --with-hdf5-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/lib/libhdf5.so --with-zlib=1 --with-zlib-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/include --with-zlib-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/lib/libz.so --with-mumps=1 --with-mumps-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/include --with-mumps-lib="/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libsmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libzmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libcmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libdmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libmumps_common.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libpord.so" --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/netlib-scalapack-2.2.0-vs6d5fhfa4i77gfo5iurpifbgxwfrmbp/lib/libscalapack.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0
[0]PETSC ERROR: #1 register_PCs() at /tmp/ulrich/spack-stage/spack-stage-tandem-main-wnfinsgrh3swjkyofitwrffzldwourlj/spack-src/app/pc/register.cpp:18

@hpc4geo
Copy link
Collaborator

hpc4geo commented Sep 1, 2023 •
edited
Loading

@Thomas-Ulrich Thomas - I believe be52dc6 (now in main) should resolve the error you encountered. The error stems from changes in PETSc and its error handler. Can you please try your test again?

@Thomas-Ulrich
Copy link
Collaborator Author

Hi Dave,
It indeed saves the first problem.

(base) ulrich@heisenbug:/export/dump/ulrich/section_8_2/scenario-rc$ mpiexec -n 2 static ridge.toml --output ridgecrest --matrix_free false --petsc -ksp_max_it 400 -pc_type jacobi -ksp_rtol 1.0e-6 -ksp_type gcr -vec_type cuda -mat_type aijcusparse -ksp_view -ksp_monitor -ksp_converged_reason -options_left -log_view

               ___          ___         _____         ___          ___
      ___     /  /\        /__/\       /  /::\       /  /\        /__/\
     /  /\   /  /::\       \  \:\     /  /:/\:\     /  /:/_      |  |::\
    /  /:/  /  /:/\:\       \  \:\   /  /:/  \:\   /  /:/ /\     |  |:|:\
   /  /:/  /  /:/~/::\  _____\__\:\ /__/:/ \__\:| /  /:/ /:/_  __|__|:|\:\
  /  /::\ /__/:/ /:/\:\/__/::::::::\\  \:\ /  /://__/:/ /:/ /\/__/::::| \:\
 /__/:/\:\\  \:\/:/__\/\  \:\~~\~~\/ \  \:\  /:/ \  \:\/:/ /:/\  \:\~~\__\/
 \__\/  \:\\  \::/      \  \:\  ~~~   \  \:\/:/   \  \::/ /:/  \  \:\
      \  \:\\  \:\       \  \:\        \  \::/     \  \:\/:/    \  \:\
       \__\/ \  \:\       \  \:\        \__\/       \  \::/      \  \:\
              \__\/        \__\/                     \__\/        \__\/

                          tandem version be52dc6

                       stack size limit = unlimited

                              Worker affinity
    0---------|----------|----------|----------|----------|----------|
    ----------|----------|----------|----------|----------|----------|
    --------8-|----------|----------|----------|----------|----------|
    ----------|----------|----------|----------|----------|----------|
    ----------|------


Ridge:create()
parsing fault_many_trace.csv
Ridge:create()
parsing fault_many_trace.csv
DOFs: 12634590
Mesh size: 36.7959
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "A" before MatSetOption_MPIAIJ()
[0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[0]PETSC ERROR: Option left: name:-ksp_max_it value: 400
[0]PETSC ERROR: Option left: name:-ksp_monitor (no value)
[0]PETSC ERROR: Option left: name:-ksp_rtol value: 1.0e-6
[0]PETSC ERROR: Option left: name:-ksp_type value: gcr
[0]PETSC ERROR: Option left: name:-ksp_view (no value)
[0]PETSC ERROR: Option left: name:-options_left (no value)
[0]PETSC ERROR: Option left: name:-pc_type value: jacobi
[0]PETSC ERROR: Option left: name:-vec_type value: cuda
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.18.5, Feb 27, 2023 
[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: Object is in wrong state
[1]PETSC ERROR: Must call MatXXXSetPreallocation(), MatSetUp() or the matrix has not yet been factored on argument 1 "A" before MatSetOption_MPIAIJ()
[1]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[1]PETSC ERROR: Option left: name:-ksp_converged_reason (no value)
[1]PETSC ERROR: Option left: name:-ksp_max_it value: 400
[1]PETSC ERROR: Option left: name:-ksp_monitor (no value)
[1]PETSC ERROR: Option left: name:-ksp_rtol value: 1.0e-6
[1]PETSC ERROR: Option left: name:-ksp_type value: gcr
[1]PETSC ERROR: Option left: name:-ksp_view (no value)
[1]PETSC ERROR: Option left: name:-options_left (no value)
[1]PETSC ERROR: Option left: name:-pc_type value: jacobi
[1]PETSC ERROR: Option left: name:-vec_type value: cuda
[1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.18.5, Feb 27, 2023 
[1]PETSC ERROR: --petsc on a  named heisenbug by ulrich Mon Sep 11 09:50:07 2023
[1]PETSC ERROR: Configure options --prefix=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 --with-make-exec=make --with-cc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpicc --with-cxx=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpic++ --with-fc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=1 --with-openmp=0 --with-64-bit-indices=1 --with-blaslapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openblas-0.3.23-kxy7bxom4lam3wx44st22oytplyuisby/lib/libopenblas.so --with-memalign=32 --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u --with-hip=0 --with-metis=1 --with-metis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/include --with-metis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/lib/libmetis.so --with-hypre=1 --with-hypre-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/include --with-hypre-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/include --with-parmetis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/lib/libparmetis.so --with-kokkos=0 --with-kokkos-kernels=0 --with-superlu_dist=1 --with-superlu_dist-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/include --with-superlu_dist-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/include --with-hdf5-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/lib/libhdf5.so --with-zlib=1 --with-zlib-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/include --with-zlib-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/lib/libz.so --with-mumps=1 --with-mumps-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/include --with-mumps-lib="/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libsmumps.so /i--petsc on a  named heisenbug by ulrich Mon Sep 11 09:50:07 2023
[0]PETSC ERROR: Configure options --prefix=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 --with-make-exec=make --with-cc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpicc --with-cxx=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpic++ --with-fc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=1 --with-openmp=0 --with-64-bit-indices=1 --with-blaslapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openblas-0.3.23-kxy7bxom4lam3wx44st22oytplyuisby/lib/libopenblas.so --with-memalign=32 --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u --with-hip=0 --with-metis=1 --with-metis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/include --with-metis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/lib/libmetis.so --with-hypre=1 --with-hypre-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/include --with-hypre-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/include --with-parmetis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/lib/libparmetis.so --with-kokkos=0 --with-kokkos-kernels=0 --with-superlu_dist=1 --with-superlu_dist-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/include --with-superlu_dist-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/include --with-hdf5-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/lib/libhdf5.so --with-zlib=1 --with-zlib-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/include --with-zlib-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/lib/libz.so --with-mumps=1 --with-mumps-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/include --with-mumps-lib="/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libsmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libzmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libcmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libdmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libmumps_common.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libpord.so" --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/netlib-scalapack-2.2.0-vs6d5fhfa4i77gfo5iurpifbgxwfrmbp/lib/libscalapack.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0
[0]PETSC ERROR: #1 MatSetOption_MPIAIJ() at /tmp/ulrich/spack-stage/spack-stage-petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah/spack-src/src/mat/impls/aij/mpi/mpiaij.c:1695
[0]PETSC ERROR: #2 MatSetOption() at /tmp/ulrich/spack-stage/spack-stage-petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah/spack-src/src/mat/interface/matrix.c:5965
[0]PETSC ERROR: #3 PetscDGMatrix() at /tmp/ulrich/spack-stage/spack-stage-tandem-main-wnfinsgrh3swjkyofitwrffzldwourlj/spack-src/app/common/PetscDGMatrix.cpp:63
terminate called after throwing an instance of 'tndm::petsc_error'
mport/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libzmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libcmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libdmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libmumps_common.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libpord.so" --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/netlib-scalapack-2.2.0-vs6d5fhfa4i77gfo5iurpifbgxwfrmbp/lib/libscalapack.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0
[1]PETSC ERROR: #1 MatSetOption_MPIAIJ() at /tmp/ulrich/spack-stage/spack-stage-petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah/spack-src/src/mat/impls/aij/mpi/mpiaij.c:1695
[1]PETSC ERROR: #2 MatSetOption() at /tmp/ulrich/spack-stage/spack-stage-petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah/spack-src/src/mat/interface/matrix.c:5965
[1]PETSC ERROR: #3 PetscDGMatrix() at /tmp/ulrich/spack-stage/spack-stage-tandem-main-wnfinsgrh3swjkyofitwrffzldwourlj/spack-src/app/common/PetscDGMatrix.cpp:63
terminate called after throwing an instance of 'tndm::petsc_error'
  what():  Object is in wrong state
[heisenbug:1769346] *** Process received signal ***
[heisenbug:1769346] Signal: Aborted (6)
[heisenbug:1769346] Associated errno: Unknown error -1 (-1)
[heisenbug:1769346] Signal code:  (1)
  what():  Object is in wrong state
[heisenbug:1769347] *** Process received signal ***
[heisenbug:1769347] Signal: Aborted (6)
[heisenbug:1769347] Signal code: User function (kill, sigsend, abort, etc.) (0)
[heisenbug:1769347] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x38d60)[0x1531cc9bfd60]
[heisenbug:1769347] [ 1] [heisenbug:1769346] [ 0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x1531cc9bfce1]
[heisenbug:1769347] [ 2] /lib/x86_64-linux-gnu/libc.so.6(+0x38d60)[0x15190e181d60]
[heisenbug:1769346] [ 1] /lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x1531cc9a9537]
[heisenbug:1769347] [ 3] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x15190e181ce1]
[heisenbug:1769346] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x15190e16b537]
[heisenbug:1769346] [ 3] /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64/libstdc++.so.6(+0xa58e3)[0x1531ccc1d8e3]
[heisenbug:1769347] [ 4] /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64/libstdc++.so.6(+0xa58e3)[0x15190e3df8e3]
[heisenbug:1769346] [ 4] /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64/libstdc++.so.6(+0xb1cda)[0x1531ccc29cda]
[heisenbug:1769347] [ 5] /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64/libstdc++.so.6(+0xb1d45)[0x1531ccc29d45]
[heisenbug:1769347] [ 6] /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64/libstdc++.so.6(+0xb1cda)[0x15190e3ebcda]
[heisenbug:1769346] [ 5] /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64/libstdc++.so.6(+0xb1ff7)[0x1531ccc29ff7]
[heisenbug:1769347] [ 7] static[0x41732b]
[heisenbug:1769347] [ 8] static[0x46f426]
[heisenbug:1769347] [ 9] static[0x445145]
[heisenbug:1769347] [10] static[0x41b9a3]
[heisenbug:1769347] [11] /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64/libstdc++.so.6(+0xb1d45)[0x15190e3ebd45]
[heisenbug:1769346] [ 6] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x1531cc9aad0a]
[heisenbug:1769347] [12] static[0x41f3ea]
[heisenbug:1769347] *** End of error message ***
/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64/libstdc++.so.6(+0xb1ff7)[0x15190e3ebff7]
[heisenbug:1769346] [ 7] static[0x41732b]
[heisenbug:1769346] [ 8] static[0x46f426]
[heisenbug:1769346] [ 9] static[0x445145]
[heisenbug:1769346] [10] static[0x41b9a3]
[heisenbug:1769346] [11] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x15190e16cd0a]
[heisenbug:1769346] [12] static[0x41f3ea]
[heisenbug:1769346] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 0 on node heisenbug exited on signal 6 (Aborted).
--------------------------------------------------------------------------

@hpc4geo
Copy link
Collaborator

hpc4geo commented Sep 13, 2023

@Thomas-Ulrich Great you ran that test and got past the immediate error. The new error is also resolved on branch dmay/petsc-gpu. Could you please re-run your test again using this branch?

@Thomas-Ulrich
Copy link
Collaborator Author

It worked!

(base) ulrich@heisenbug:/export/dump/ulrich/section_8_2/scenario-rc$ mpiexec -n 2 static ridge.toml --output ridgecrest --matrix_free false --petsc -ksp_max_it 400 -pc_type jacobi -ksp_rtol 1.0e-6 -ksp_type gcr -vec_type cuda -mat_type aijcusparse -ksp_view -ksp_monitor -ksp_converged_reason -options_left -log_view

               ___          ___         _____         ___          ___
      ___     /  /\        /__/\       /  /::\       /  /\        /__/\
     /  /\   /  /::\       \  \:\     /  /:/\:\     /  /:/_      |  |::\
    /  /:/  /  /:/\:\       \  \:\   /  /:/  \:\   /  /:/ /\     |  |:|:\
   /  /:/  /  /:/~/::\  _____\__\:\ /__/:/ \__\:| /  /:/ /:/_  __|__|:|\:\
  /  /::\ /__/:/ /:/\:\/__/::::::::\\  \:\ /  /://__/:/ /:/ /\/__/::::| \:\
 /__/:/\:\\  \:\/:/__\/\  \:\~~\~~\/ \  \:\  /:/ \  \:\/:/ /:/\  \:\~~\__\/
 \__\/  \:\\  \::/      \  \:\  ~~~   \  \:\/:/   \  \::/ /:/  \  \:\
      \  \:\\  \:\       \  \:\        \  \::/     \  \:\/:/    \  \:\
       \__\/ \  \:\       \  \:\        \__\/       \  \::/      \  \:\
              \__\/        \__\/                     \__\/        \__\/

                          tandem version 4b6c45e

                       stack size limit = unlimited

                              Worker affinity
    0---------|----------|----------|----------|----------|----------|
    ----------|----------|----------|----------|----------|----------|
    --------8-|----------|----------|----------|----------|----------|
    ----------|----------|----------|----------|----------|----------|
    ----------|------


Ridge:create()
parsing fault_many_trace.csv
Ridge:create()
parsing fault_many_trace.csv
DOFs: 12634590
Mesh size: 36.7959
Assembly: 236.758 s
Solver warmup: 3.39949 s
  0 KSP Residual norm 1.043596231397e+04 
  1 KSP Residual norm 5.057045424157e+03 
  2 KSP Residual norm 2.501262926538e+03 
  3 KSP Residual norm 1.404021748561e+03 
  4 KSP Residual norm 9.405595891919e+02 
  5 KSP Residual norm 6.255057299835e+02 
  6 KSP Residual norm 4.470847627376e+02 
  7 KSP Residual norm 3.490249256949e+02 
  8 KSP Residual norm 2.809121341066e+02 
  9 KSP Residual norm 2.221057145177e+02 
 10 KSP Residual norm 1.748130469702e+02 
 11 KSP Residual norm 1.401902300234e+02 
 12 KSP Residual norm 1.164758176460e+02 
 13 KSP Residual norm 9.867590915172e+01 
 14 KSP Residual norm 8.539765892122e+01 
 15 KSP Residual norm 7.650900132972e+01 
 16 KSP Residual norm 7.000515989883e+01 
 17 KSP Residual norm 6.498017121417e+01 
 18 KSP Residual norm 6.110665939969e+01 
 19 KSP Residual norm 5.791056463201e+01 
 20 KSP Residual norm 5.505070821390e+01 
 21 KSP Residual norm 5.226404388711e+01 
 22 KSP Residual norm 4.943990647117e+01 
 23 KSP Residual norm 4.665549043271e+01 
 24 KSP Residual norm 4.399544536644e+01 
 25 KSP Residual norm 4.154164239417e+01 
 26 KSP Residual norm 3.939470141669e+01 
 27 KSP Residual norm 3.756470281537e+01 
 28 KSP Residual norm 3.598582603206e+01 
 29 KSP Residual norm 3.464281664502e+01 
 30 KSP Residual norm 3.347478047406e+01 
 31 KSP Residual norm 3.297684428650e+01 
 32 KSP Residual norm 3.258314495897e+01 
 33 KSP Residual norm 3.198439669474e+01 
 34 KSP Residual norm 3.125069693726e+01 
 35 KSP Residual norm 3.050668986486e+01 
 36 KSP Residual norm 2.968574246339e+01 
 37 KSP Residual norm 2.880077966912e+01 
 38 KSP Residual norm 2.804585632615e+01 
 39 KSP Residual norm 2.729940854350e+01 
 40 KSP Residual norm 2.643360874144e+01 
 41 KSP Residual norm 2.565851385890e+01 
 42 KSP Residual norm 2.505383388313e+01 
 43 KSP Residual norm 2.445484086684e+01 
 44 KSP Residual norm 2.353615935008e+01 
 45 KSP Residual norm 2.275423567003e+01 
 46 KSP Residual norm 2.176545206889e+01 
 47 KSP Residual norm 2.080768417124e+01 
 48 KSP Residual norm 2.001738877707e+01 
 49 KSP Residual norm 1.913086340748e+01 
 50 KSP Residual norm 1.827692355724e+01 
 51 KSP Residual norm 1.768092737476e+01 
 52 KSP Residual norm 1.717450697766e+01 
 53 KSP Residual norm 1.668094961826e+01 
 54 KSP Residual norm 1.621022411943e+01 
 55 KSP Residual norm 1.575309194028e+01 
 56 KSP Residual norm 1.539103535047e+01 
 57 KSP Residual norm 1.512056339802e+01 
 58 KSP Residual norm 1.481764991587e+01 
 59 KSP Residual norm 1.458155833150e+01 
 60 KSP Residual norm 1.429919323552e+01 
 61 KSP Residual norm 1.415517037049e+01 
 62 KSP Residual norm 1.399234017229e+01 
 63 KSP Residual norm 1.378311668948e+01 
 64 KSP Residual norm 1.361947706540e+01 
 65 KSP Residual norm 1.344335545506e+01 
 66 KSP Residual norm 1.326532197942e+01 
 67 KSP Residual norm 1.309807648346e+01 
 68 KSP Residual norm 1.296657427835e+01 
 69 KSP Residual norm 1.283695108698e+01 
 70 KSP Residual norm 1.267007359103e+01 
 71 KSP Residual norm 1.244710456209e+01 
 72 KSP Residual norm 1.220370234429e+01 
 73 KSP Residual norm 1.203832481144e+01 
 74 KSP Residual norm 1.187122962688e+01 
 75 KSP Residual norm 1.170152324190e+01 
 76 KSP Residual norm 1.153462959335e+01 
 77 KSP Residual norm 1.134697897958e+01 
 78 KSP Residual norm 1.117215824238e+01 
 79 KSP Residual norm 1.101994065827e+01 
 80 KSP Residual norm 1.086476019357e+01 
 81 KSP Residual norm 1.070942975098e+01 
 82 KSP Residual norm 1.052267996890e+01 
 83 KSP Residual norm 1.034052442724e+01 
 84 KSP Residual norm 1.021009119411e+01 
 85 KSP Residual norm 1.010111395955e+01 
 86 KSP Residual norm 9.999392332292e+00 
 87 KSP Residual norm 9.905773804438e+00 
 88 KSP Residual norm 9.801442737859e+00 
 89 KSP Residual norm 9.677046033847e+00 
 90 KSP Residual norm 9.556033128703e+00 
 91 KSP Residual norm 9.478547429629e+00 
 92 KSP Residual norm 9.407015314605e+00 
 93 KSP Residual norm 9.333198724367e+00 
 94 KSP Residual norm 9.256767561751e+00 
 95 KSP Residual norm 9.190387305344e+00 
 96 KSP Residual norm 9.126105862277e+00 
 97 KSP Residual norm 9.065686270195e+00 
 98 KSP Residual norm 8.999945759040e+00 
 99 KSP Residual norm 8.926154965140e+00 
100 KSP Residual norm 8.849610367048e+00 
101 KSP Residual norm 8.789846778250e+00 
102 KSP Residual norm 8.741873464051e+00 
103 KSP Residual norm 8.682880521869e+00 
104 KSP Residual norm 8.608840997956e+00 
105 KSP Residual norm 8.538616256590e+00 
106 KSP Residual norm 8.447613366794e+00 
107 KSP Residual norm 8.369318452434e+00 
108 KSP Residual norm 8.293197836615e+00 
109 KSP Residual norm 8.186303860330e+00 
110 KSP Residual norm 8.080866637053e+00 
111 KSP Residual norm 8.009963000677e+00 
112 KSP Residual norm 7.936414858358e+00 
113 KSP Residual norm 7.858744503449e+00 
114 KSP Residual norm 7.785325044790e+00 
115 KSP Residual norm 7.718060802479e+00 
116 KSP Residual norm 7.652779344931e+00 
117 KSP Residual norm 7.597380224417e+00 
118 KSP Residual norm 7.537225446873e+00 
119 KSP Residual norm 7.483421623934e+00 
120 KSP Residual norm 7.423911804535e+00 
121 KSP Residual norm 7.379583814341e+00 
122 KSP Residual norm 7.331324777632e+00 
123 KSP Residual norm 7.272492000865e+00 
124 KSP Residual norm 7.222459053248e+00 
125 KSP Residual norm 7.172445329407e+00 
126 KSP Residual norm 7.131041426961e+00 
127 KSP Residual norm 7.096845311950e+00 
128 KSP Residual norm 7.068263965019e+00 
129 KSP Residual norm 7.041589260996e+00 
130 KSP Residual norm 7.007516785897e+00 
131 KSP Residual norm 6.953625587450e+00 
132 KSP Residual norm 6.898105685477e+00 
133 KSP Residual norm 6.861636296388e+00 
134 KSP Residual norm 6.825263405788e+00 
135 KSP Residual norm 6.782291761785e+00 
136 KSP Residual norm 6.741969965335e+00 
137 KSP Residual norm 6.692448746921e+00 
138 KSP Residual norm 6.638196566239e+00 
139 KSP Residual norm 6.593128517003e+00 
140 KSP Residual norm 6.550147749365e+00 
141 KSP Residual norm 6.493865509721e+00 
142 KSP Residual norm 6.425750033650e+00 
143 KSP Residual norm 6.361853225145e+00 
144 KSP Residual norm 6.315089040734e+00 
145 KSP Residual norm 6.279686250644e+00 
146 KSP Residual norm 6.250052873209e+00 
147 KSP Residual norm 6.222320326304e+00 
148 KSP Residual norm 6.188458221744e+00 
149 KSP Residual norm 6.141144743962e+00 
150 KSP Residual norm 6.096183275765e+00 
151 KSP Residual norm 6.063027506216e+00 
152 KSP Residual norm 6.032543296161e+00 
153 KSP Residual norm 6.005497369794e+00 
154 KSP Residual norm 5.978094977646e+00 
155 KSP Residual norm 5.954260349365e+00 
156 KSP Residual norm 5.929271521578e+00 
157 KSP Residual norm 5.905501967380e+00 
158 KSP Residual norm 5.878640338841e+00 
159 KSP Residual norm 5.845898054536e+00 
160 KSP Residual norm 5.814149282054e+00 
161 KSP Residual norm 5.793082821883e+00 
162 KSP Residual norm 5.775315899151e+00 
163 KSP Residual norm 5.750861401791e+00 
164 KSP Residual norm 5.721078852141e+00 
165 KSP Residual norm 5.696176634816e+00 
166 KSP Residual norm 5.662368413063e+00 
167 KSP Residual norm 5.633670555778e+00 
168 KSP Residual norm 5.606703346582e+00 
169 KSP Residual norm 5.562363039072e+00 
170 KSP Residual norm 5.519887153291e+00 
171 KSP Residual norm 5.497221286684e+00 
172 KSP Residual norm 5.474307358870e+00 
173 KSP Residual norm 5.442147697828e+00 
174 KSP Residual norm 5.414935999541e+00 
175 KSP Residual norm 5.389603003780e+00 
176 KSP Residual norm 5.359257652197e+00 
177 KSP Residual norm 5.332741202265e+00 
178 KSP Residual norm 5.306101555765e+00 
179 KSP Residual norm 5.279035487319e+00 
180 KSP Residual norm 5.246320139231e+00 
181 KSP Residual norm 5.220859567258e+00 
182 KSP Residual norm 5.197090958094e+00 
183 KSP Residual norm 5.170997760256e+00 
184 KSP Residual norm 5.147418048527e+00 
185 KSP Residual norm 5.122829552154e+00 
186 KSP Residual norm 5.104238656476e+00 
187 KSP Residual norm 5.088648018557e+00 
188 KSP Residual norm 5.074672467950e+00 
189 KSP Residual norm 5.064078259071e+00 
190 KSP Residual norm 5.051134957237e+00 
191 KSP Residual norm 5.026408331782e+00 
192 KSP Residual norm 5.001387802286e+00 
193 KSP Residual norm 4.987260410521e+00 
194 KSP Residual norm 4.972466261626e+00 
195 KSP Residual norm 4.952317170538e+00 
196 KSP Residual norm 4.933161328568e+00 
197 KSP Residual norm 4.908199410388e+00 
198 KSP Residual norm 4.881668844609e+00 
199 KSP Residual norm 4.859451470171e+00 
200 KSP Residual norm 4.837492073040e+00 
201 KSP Residual norm 4.802548131604e+00 
202 KSP Residual norm 4.763116268308e+00 
203 KSP Residual norm 4.729593735448e+00 
204 KSP Residual norm 4.702323341804e+00 
205 KSP Residual norm 4.682236020272e+00 
206 KSP Residual norm 4.667127557911e+00 
207 KSP Residual norm 4.653891837228e+00 
208 KSP Residual norm 4.637050314339e+00 
209 KSP Residual norm 4.611100460951e+00 
210 KSP Residual norm 4.586613371416e+00 
211 KSP Residual norm 4.567567622865e+00 
212 KSP Residual norm 4.550257380318e+00 
213 KSP Residual norm 4.535396817447e+00 
214 KSP Residual norm 4.520708652979e+00 
215 KSP Residual norm 4.506936326780e+00 
216 KSP Residual norm 4.491583422005e+00 
217 KSP Residual norm 4.476983996457e+00 
218 KSP Residual norm 4.462243678662e+00 
219 KSP Residual norm 4.442670098889e+00 
220 KSP Residual norm 4.421820770732e+00 
221 KSP Residual norm 4.409589159754e+00 
222 KSP Residual norm 4.399408053243e+00 
223 KSP Residual norm 4.385784857156e+00 
224 KSP Residual norm 4.367992808661e+00 
225 KSP Residual norm 4.354092605304e+00 
226 KSP Residual norm 4.335595592299e+00 
227 KSP Residual norm 4.320547006262e+00 
228 KSP Residual norm 4.307546157350e+00 
229 KSP Residual norm 4.284966186583e+00 
230 KSP Residual norm 4.265524634571e+00 
231 KSP Residual norm 4.254219021495e+00 
232 KSP Residual norm 4.241769749282e+00 
233 KSP Residual norm 4.224541635452e+00 
234 KSP Residual norm 4.212076794694e+00 
235 KSP Residual norm 4.199364068099e+00 
236 KSP Residual norm 4.182420122043e+00 
237 KSP Residual norm 4.167392512806e+00 
238 KSP Residual norm 4.149258317056e+00 
239 KSP Residual norm 4.134252028716e+00 
240 KSP Residual norm 4.114890232435e+00 
241 KSP Residual norm 4.099629659110e+00 
242 KSP Residual norm 4.086309475062e+00 
243 KSP Residual norm 4.068164064749e+00 
244 KSP Residual norm 4.053303344208e+00 
245 KSP Residual norm 4.038519239985e+00 
246 KSP Residual norm 4.029588912426e+00 
247 KSP Residual norm 4.022566445018e+00 
248 KSP Residual norm 4.015229952986e+00 
249 KSP Residual norm 4.008435017122e+00 
250 KSP Residual norm 4.001202566444e+00 
251 KSP Residual norm 3.989624635887e+00 
252 KSP Residual norm 3.976891273831e+00 
253 KSP Residual norm 3.968458076782e+00 
254 KSP Residual norm 3.959786812362e+00 
255 KSP Residual norm 3.948659177770e+00 
256 KSP Residual norm 3.937139609303e+00 
257 KSP Residual norm 3.923237527944e+00 
258 KSP Residual norm 3.909421728740e+00 
259 KSP Residual norm 3.895462326455e+00 
260 KSP Residual norm 3.879782111884e+00 
261 KSP Residual norm 3.857084779396e+00 
262 KSP Residual norm 3.834156877016e+00 
263 KSP Residual norm 3.812373918671e+00 
264 KSP Residual norm 3.791430954743e+00 
265 KSP Residual norm 3.776323581593e+00 
266 KSP Residual norm 3.764956376798e+00 
267 KSP Residual norm 3.755674485260e+00 
268 KSP Residual norm 3.746805121124e+00 
269 KSP Residual norm 3.732410498860e+00 
270 KSP Residual norm 3.716334071608e+00 
271 KSP Residual norm 3.704603927799e+00 
272 KSP Residual norm 3.693963613260e+00 
273 KSP Residual norm 3.685913887226e+00 
274 KSP Residual norm 3.675983155710e+00 
275 KSP Residual norm 3.666747603813e+00 
276 KSP Residual norm 3.655282991832e+00 
277 KSP Residual norm 3.643299078303e+00 
278 KSP Residual norm 3.632794313968e+00 
279 KSP Residual norm 3.620066540668e+00 
280 KSP Residual norm 3.604988051275e+00 
281 KSP Residual norm 3.595573325810e+00 
282 KSP Residual norm 3.589074695633e+00 
283 KSP Residual norm 3.580612407923e+00 
284 KSP Residual norm 3.569280200629e+00 
285 KSP Residual norm 3.559112180224e+00 
286 KSP Residual norm 3.547885161717e+00 
287 KSP Residual norm 3.538750379302e+00 
288 KSP Residual norm 3.529913444267e+00 
289 KSP Residual norm 3.517916190869e+00 
290 KSP Residual norm 3.506831629967e+00 
291 KSP Residual norm 3.499628471422e+00 
292 KSP Residual norm 3.491947274112e+00 
293 KSP Residual norm 3.483867470911e+00 
294 KSP Residual norm 3.477023496516e+00 
295 KSP Residual norm 3.469299394561e+00 
296 KSP Residual norm 3.458304509888e+00 
297 KSP Residual norm 3.448524871219e+00 
298 KSP Residual norm 3.435541423956e+00 
299 KSP Residual norm 3.426325791695e+00 
300 KSP Residual norm 3.415704180855e+00 
301 KSP Residual norm 3.406956378580e+00 
302 KSP Residual norm 3.398157213870e+00 
303 KSP Residual norm 3.385176233993e+00 
304 KSP Residual norm 3.376231299875e+00 
305 KSP Residual norm 3.367346162844e+00 
306 KSP Residual norm 3.361408504911e+00 
307 KSP Residual norm 3.357002963659e+00 
308 KSP Residual norm 3.353052451083e+00 
309 KSP Residual norm 3.348588875229e+00 
310 KSP Residual norm 3.343814464155e+00 
311 KSP Residual norm 3.336636510074e+00 
312 KSP Residual norm 3.329340373091e+00 
313 KSP Residual norm 3.323882913378e+00 
314 KSP Residual norm 3.318333421839e+00 
315 KSP Residual norm 3.311670969810e+00 
316 KSP Residual norm 3.303866081683e+00 
317 KSP Residual norm 3.293903325053e+00 
318 KSP Residual norm 3.284415370212e+00 
319 KSP Residual norm 3.276445522193e+00 
320 KSP Residual norm 3.265088857299e+00 
321 KSP Residual norm 3.246017818685e+00 
322 KSP Residual norm 3.228557958164e+00 
323 KSP Residual norm 3.213326763456e+00 
324 KSP Residual norm 3.197662546060e+00 
325 KSP Residual norm 3.186347625444e+00 
326 KSP Residual norm 3.178750308424e+00 
327 KSP Residual norm 3.172216453971e+00 
328 KSP Residual norm 3.165281508617e+00 
329 KSP Residual norm 3.154641178174e+00 
330 KSP Residual norm 3.144982566200e+00 
331 KSP Residual norm 3.137119791603e+00 
332 KSP Residual norm 3.129246608571e+00 
333 KSP Residual norm 3.122992707017e+00 
334 KSP Residual norm 3.115608713024e+00 
335 KSP Residual norm 3.109089926533e+00 
336 KSP Residual norm 3.100374442263e+00 
337 KSP Residual norm 3.090489356646e+00 
338 KSP Residual norm 3.082110045784e+00 
339 KSP Residual norm 3.071728903912e+00 
340 KSP Residual norm 3.059906974716e+00 
341 KSP Residual norm 3.052784156457e+00 
342 KSP Residual norm 3.048278027309e+00 
343 KSP Residual norm 3.041593216215e+00 
344 KSP Residual norm 3.032786260029e+00 
345 KSP Residual norm 3.025747098684e+00 
346 KSP Residual norm 3.018450997932e+00 
347 KSP Residual norm 3.012318935933e+00 
348 KSP Residual norm 3.006302956491e+00 
349 KSP Residual norm 2.998708362493e+00 
350 KSP Residual norm 2.991714324831e+00 
351 KSP Residual norm 2.986781453606e+00 
352 KSP Residual norm 2.982059693684e+00 
353 KSP Residual norm 2.976779119791e+00 
354 KSP Residual norm 2.972556574587e+00 
355 KSP Residual norm 2.967460106369e+00 
356 KSP Residual norm 2.959862792046e+00 
357 KSP Residual norm 2.952554334154e+00 
358 KSP Residual norm 2.944615992893e+00 
359 KSP Residual norm 2.937923253544e+00 
360 KSP Residual norm 2.930246358250e+00 
361 KSP Residual norm 2.923790136805e+00 
362 KSP Residual norm 2.917473890658e+00 
363 KSP Residual norm 2.909686398104e+00 
364 KSP Residual norm 2.902549932091e+00 
365 KSP Residual norm 2.895930952786e+00 
366 KSP Residual norm 2.891807691300e+00 
367 KSP Residual norm 2.889196349786e+00 
368 KSP Residual norm 2.886384509164e+00 
369 KSP Residual norm 2.883553930330e+00 
370 KSP Residual norm 2.880239285194e+00 
371 KSP Residual norm 2.875691771212e+00 
372 KSP Residual norm 2.870633791727e+00 
373 KSP Residual norm 2.866884219455e+00 
374 KSP Residual norm 2.862822558442e+00 
375 KSP Residual norm 2.858089817529e+00 
376 KSP Residual norm 2.853032859178e+00 
377 KSP Residual norm 2.845858835949e+00 
378 KSP Residual norm 2.838521568686e+00 
379 KSP Residual norm 2.832934327027e+00 
380 KSP Residual norm 2.824901796916e+00 
381 KSP Residual norm 2.811610040249e+00 
382 KSP Residual norm 2.796171157330e+00 
383 KSP Residual norm 2.783463007661e+00 
384 KSP Residual norm 2.770689943351e+00 
385 KSP Residual norm 2.761879288812e+00 
386 KSP Residual norm 2.756281179635e+00 
387 KSP Residual norm 2.751514098498e+00 
388 KSP Residual norm 2.745807044433e+00 
389 KSP Residual norm 2.737994732321e+00 
390 KSP Residual norm 2.730991568069e+00 
391 KSP Residual norm 2.725330722286e+00 
392 KSP Residual norm 2.719405115155e+00 
393 KSP Residual norm 2.714264846840e+00 
394 KSP Residual norm 2.708919391119e+00 
395 KSP Residual norm 2.703866634930e+00 
396 KSP Residual norm 2.696704213343e+00 
397 KSP Residual norm 2.688371849182e+00 
398 KSP Residual norm 2.680824401227e+00 
399 KSP Residual norm 2.671782610951e+00 
400 KSP Residual norm 2.662524934212e+00 
Linear solve converged due to CONVERGED_ITS iterations 400
KSP Object: 2 MPI processes
  type: gcr
    restart = 30 
    restarts performed = 14 
  maximum iterations=400, initial guess is zero
  tolerances:  relative=1e-06, absolute=1e-50, divergence=10000.
  right preconditioning
  using UNPRECONDITIONED norm type for convergence test
PC Object: 2 MPI processes
  type: jacobi
    type DIAGONAL
  linear system matrix = precond matrix:
  Mat Object: 2 MPI processes
    type: mpiaijcusparse
    rows=12634590, cols=12634590, bs=30
    total: nonzeros=1886343300, allocated nonzeros=1886343300
    total number of mallocs used during MatSetValues calls=0
      not using I-node (on process 0) routines
Solve: 65.892 s
Residual norm: 2.66252
Iterations: 400
****************************************************************************************************************************************************************
***                                WIDEN YOUR WINDOW TO 160 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document                                 ***
****************************************************************************************************************************************************************

------------------------------------------------------------------ PETSc Performance Summary: ------------------------------------------------------------------



      ##########################################################
      #                                                        #
      #                       WARNING!!!                       #
      #                                                        #
      #   This code was compiled with a debugging option.      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################


--petsc on a  named heisenbug with 2 processors, by ulrich Wed Sep 13 21:43:59 2023
Using Petsc Release Version 3.18.5, Feb 27, 2023 

                         Max       Max/Min     Avg       Total
Time (sec):           3.338e+02     1.000   3.338e+02
Objects:              8.200e+01     1.000   8.200e+01
Flops:                1.028e+12     1.051   1.004e+12  2.007e+12
Flops/sec:            3.081e+09     1.051   3.007e+09  6.013e+09
Memory (bytes):       1.608e+10     1.050   1.570e+10  3.139e+10
MPI Msg Count:        4.030e+02     1.000   4.030e+02  8.060e+02
MPI Msg Len (bytes):  1.868e+08     1.000   4.635e+05  3.736e+08
MPI Reductions:       3.103e+04     1.000

Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N --> 2N flops
                            and VecAXPY() for complex vectors of length N --> 8N flops

Summary of Stages:   ----- Time ------  ----- Flop ------  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total    Count   %Total     Avg         %Total    Count   %Total
 0:      Main Stage: 2.6791e+02  80.3%  1.5075e+09   0.1%  0.000e+00   0.0%  0.000e+00        0.0%  2.100e+02   0.7%
 2:           solve: 6.5892e+01  19.7%  2.0056e+12  99.9%  8.060e+02 100.0%  4.635e+05      100.0%  3.080e+04  99.3%

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flop: Max - maximum over all processors
                  Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   AvgLen: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
      %T - percent time in this phase         %F - percent flop in this phase
      %M - percent messages in this phase     %L - percent message lengths in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors)
   GPU Mflop/s: 10e-6 * (sum of flop on GPU over all processors)/(max GPU time over all processors)
   CpuToGpu Count: total number of CPU to GPU copies per processor
   CpuToGpu Size (Mbytes): 10e-6 * (total size of CPU to GPU copies per processor)
   GpuToCpu Count: total number of GPU to CPU copies per processor
   GpuToCpu Size (Mbytes): 10e-6 * (total size of GPU to CPU copies per processor)
   GPU %F: percent flops on GPU in this event
------------------------------------------------------------------------------------------------------------------------


      ##########################################################
      #                                                        #
      #                       WARNING!!!                       #
      #                                                        #
      #   This code was compiled with a debugging option.      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################


Event                Count      Time (sec)     Flop                              --- Global ---  --- Stage ----  Total   GPU    - CpuToGpu -   - GpuToCpu - GPU
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   AvgLen  Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s Mflop/s Count   Size   Count   Size  %F
---------------------------------------------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

BuildTwoSided          1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  1  -nan    -nan      0 0.00e+00    0 0.00e+00  0
BuildTwoSidedF         1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  1  -nan    -nan      0 0.00e+00    0 0.00e+00  0
PCSetUp                1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
MatAssemblyBegin       1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  1  0  0  0  0   1  0  0  0  2  -nan    -nan      0 0.00e+00    0 0.00e+00  0
MatAssemblyEnd         1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 1.9e+01  1  0  0  0  0   1  0  0  0  9  -nan    -nan      0 0.00e+00    0 0.00e+00  0
VecSet                63 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
VecCUDACopyFrom        1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    1 5.05e+01  0
SFSetGraph             1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
KSPSetUp               1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 1.3e+02  1  0  0  0  0   1  0  0  0 61  -nan    -nan      0 0.00e+00    0 0.00e+00  0

--- Event Stage 1: Unknown


--- Event Stage 2: solve

BuildTwoSided          1 1.0   nan nan 0.00e+00 0.0 2.0e+00 8.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
PCApply              400 1.0   nan nan 2.59e+09 1.1 0.0e+00 0.0e+00 9.0e+00  1  0  0  0  0   3  0  0  0  0  -nan    -nan      2 1.01e+02    1 5.05e+01  100
MatMult              401 1.0   nan nan 7.73e+11 1.1 8.1e+02 4.6e+05 2.0e+00 17 75 100 100  0  87 75 100 100  0  -nan    -nan      2 1.13e+04    0 0.00e+00  100
MatView                1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
MatCUSPARSCopyTo       2 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 14  0  0  0  0  71  0  0  0  0  -nan    -nan      2 1.13e+04    0 0.00e+00  0
cuBLAS Init            1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   5  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
DCtxCreate             1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
DCtxSetUp              1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
DCtxSetDevice          1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
VecDotNorm2          400 1.0   nan nan 1.04e+10 1.1 0.0e+00 0.0e+00 8.0e+02  1  1  0  0  3   3  1  0  0  3  -nan    -nan      0 0.00e+00    0 0.00e+00  100
VecMDot              386 1.0   nan nan 7.38e+10 1.1 0.0e+00 0.0e+00 7.7e+02  0  7  0  0  2   1  7  0  0  3  -nan    -nan      0 0.00e+00    0 0.00e+00  100
VecNorm              401 1.0   nan nan 5.19e+09 1.1 0.0e+00 0.0e+00 8.0e+02  0  1  0  0  3   0  1  0  0  3  -nan    -nan      0 0.00e+00    0 0.00e+00  100
VecScale             800 1.0   nan nan 5.18e+09 1.1 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  100
VecSet                 2 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
VecAXPY              800 1.0   nan nan 1.04e+10 1.1 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  100
VecAYPX                1 1.0   nan nan 1.30e+07 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      1 5.05e+01    0 0.00e+00  100
VecMAXPY             772 1.0   nan nan 1.48e+11 1.1 0.0e+00 0.0e+00 0.0e+00  0 14  0  0  0   0 14  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  100
VecPointwiseMult     400 1.0   nan nan 2.59e+09 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      1 5.05e+01    0 0.00e+00  100
VecScatterBegin      401 1.0   nan nan 0.00e+00 0.0 8.1e+02 4.6e+05 2.0e+00  0  0 100 100  0   0  0 100 100  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
VecScatterEnd        401 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  3  0  0  0  0  16  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
VecCUDACopyTo          3 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      3 1.52e+02    0 0.00e+00  0
VecCUDACopyFrom        1 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    1 5.05e+01  0
SFSetUp                1 1.0   nan nan 0.00e+00 0.0 4.0e+00 2.3e+05 2.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
SFPack               401 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
SFUnpack             401 1.0   nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  -nan    -nan      0 0.00e+00    0 0.00e+00  0
KSPSolve               1 1.0 6.5892e+01 1.0 1.03e+12 1.1 8.1e+02 4.6e+05 3.1e+04 20 100 100 100 99 100 100 100 100 100 30438    -nan      5 1.15e+04    1 5.05e+01  100
---------------------------------------------------------------------------------------------------------------------------------------------------------------

Object Type          Creations   Destructions. Reports information only for process 0.

--- Event Stage 0: Main Stage

           Container     0              3
      Preconditioner     1              1
              Matrix     3              3
           Index Set     2              2
   IS L to G Mapping     2              2
              Vector    65             66
   Star Forest Graph     1              1
       Krylov Solver     1              1
              Viewer     2              1

--- Event Stage 1: Unknown


--- Event Stage 2: solve

           Container     3              0
  PetscDeviceContext     1              0
              Vector     1              0
========================================================================================================================
Average time to get PetscTime(): 3.7e-08
Average time for MPI_Barrier(): 1.09678e-05
Average time for zero size MPI_Send(): 1.772e-05
#PETSc Option Table entries:
-ksp_converged_reason
-ksp_max_it 400
-ksp_monitor
-ksp_rtol 1.0e-6
-ksp_type gcr
-ksp_view
-log_view
-mat_type aijcusparse
-options_left
-pc_type jacobi
-vec_type cuda
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with 64 bit PetscInt
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8
Configure options: --prefix=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah --with-ssl=0 --download-c2html=0 --download-sowing=0 --download-hwloc=0 --with-make-exec=make --with-cc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpicc --with-cxx=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpic++ --with-fc=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpif90 --with-precision=double --with-scalar-type=real --with-shared-libraries=1 --with-debugging=1 --with-openmp=0 --with-64-bit-indices=1 --with-blaslapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openblas-0.3.23-kxy7bxom4lam3wx44st22oytplyuisby/lib/libopenblas.so --with-memalign=32 --with-x=0 --with-clanguage=C --with-cuda=1 --with-cuda-dir=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u --with-hip=0 --with-metis=1 --with-metis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/include --with-metis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/lib/libmetis.so --with-hypre=1 --with-hypre-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/include --with-hypre-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/lib/libHYPRE.so --with-parmetis=1 --with-parmetis-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/include --with-parmetis-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/lib/libparmetis.so --with-kokkos=0 --with-kokkos-kernels=0 --with-superlu_dist=1 --with-superlu_dist-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/include --with-superlu_dist-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/lib/libsuperlu_dist.so --with-ptscotch=0 --with-suitesparse=0 --with-hdf5=1 --with-hdf5-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/include --with-hdf5-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/lib/libhdf5.so --with-zlib=1 --with-zlib-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/include --with-zlib-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/lib/libz.so --with-mumps=1 --with-mumps-include=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/include --with-mumps-lib="/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libsmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libzmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libcmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libdmumps.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libmumps_common.so /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib/libpord.so" --with-trilinos=0 --with-fftw=0 --with-valgrind=0 --with-gmp=0 --with-libpng=0 --with-giflib=0 --with-mpfr=0 --with-netcdf=0 --with-pnetcdf=0 --with-moab=0 --with-random123=0 --with-exodusii=0 --with-cgns=0 --with-memkind=0 --with-p4est=0 --with-saws=0 --with-yaml=0 --with-hwloc=0 --with-libjpeg=0 --with-scalapack=1 --with-scalapack-lib=/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/netlib-scalapack-2.2.0-vs6d5fhfa4i77gfo5iurpifbgxwfrmbp/lib/libscalapack.so --with-strumpack=0 --with-mmg=0 --with-parmmg=0 --with-tetgen=0
-----------------------------------------
Libraries compiled on 2023-08-31 16:19:23 on heisenbug 
Machine characteristics: Linux-6.1.0-0.deb11.7-amd64-x86_64-with-glibc2.31
Using PETSc directory: /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah
Using PETSc arch: 
-----------------------------------------

Using C compiler: /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpicc  -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g3 -O0   
Using Fortran compiler: /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpif90  -fPIC -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -g -O0     
-----------------------------------------

Using include paths: -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah/include -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/include -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/include -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/include -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/include -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/include -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/include -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/include -I/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u/include
-----------------------------------------

Using C linker: /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpicc
Using Fortran linker: /import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/bin/mpif90
Using libraries: -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/petsc-3.18.5-pppgzrc7ykm26yk4dmy7nzuf2f6jopah/lib -lpetsc -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hypre-2.28.0-73iiib7speczh6bpszpkvbixgpkwyt7z/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/mumps-5.5.1-umzads25ggryhklmvvto4a5i6ue74fbk/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/netlib-scalapack-2.2.0-vs6d5fhfa4i77gfo5iurpifbgxwfrmbp/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/netlib-scalapack-2.2.0-vs6d5fhfa4i77gfo5iurpifbgxwfrmbp/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/superlu-dist-8.1.2-7mrjdpdpu5l2g7x546r4kcssv2minjsa/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openblas-0.3.23-kxy7bxom4lam3wx44st22oytplyuisby/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openblas-0.3.23-kxy7bxom4lam3wx44st22oytplyuisby/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hdf5-1.14.2-dstkldnhuuoqsxniwh7piimjbyva4b2c/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/parmetis-4.0.3-gy6mgt3a4qt2ppnsorma4tabjw3f55fw/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/metis-5.1.0-6zrgzhlze5ujsajtktgwf73dlpdjtdxk/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/zlib-1.2.13-ufda23qzqsm5z2eqwsne76yoazck6fsz/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u/lib64 -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u/lib64 -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/cuda-11.8.0-uivzrqah34rk27dutzqehubyc57zi46u/lib64/stubs -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/openmpi-4.1.5-sf7iuo3braeberbmnnyt4emapkxz4jtw/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hwloc-2.9.1-sm5zkzc5cbvuw6us4oahy5bg6ygp5goi/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/hwloc-2.9.1-sm5zkzc5cbvuw6us4oahy5bg6ygp5goi/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/libevent-2.1.12-7xvtpwwyxrysr7sq5atthy7hekevfifs/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/libevent-2.1.12-7xvtpwwyxrysr7sq5atthy7hekevfifs/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/pmix-4.2.4-l3hzxh27fe2svyq3tkwykk744kvdunps/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-11.2.0/pmix-4.2.4-l3hzxh27fe2svyq3tkwykk744kvdunps/lib -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64 -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib/gcc/x86_64-pc-linux-gnu/11.2.0 -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib/gcc/x86_64-pc-linux-gnu/11.2.0 -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib64 -Wl,-rpath,/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib -L/import/exception-dump/ulrich/myLibs/spack-packages/linux-debian11-zen2/gcc-12.2.0/gcc-11.2.0-iokm55jhztsmmlbgy3mr5rvu3y43qxyz/lib -lHYPRE -lsmumps -lzmumps -lcmumps -ldmumps -lmumps_common -lpord -lscalapack -lsuperlu_dist -lopenblas -lhdf5 -lparmetis -lmetis -lm -lz -lcudart -lnvToolsExt -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lquadmath -lstdc++ -ldl
-----------------------------------------



      ##########################################################
      #                                                        #
      #                       WARNING!!!                       #
      #                                                        #
      #   This code was compiled with a debugging option.      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################


#PETSc Option Table entries:
-ksp_converged_reason
-ksp_max_it 400
-ksp_monitor
-ksp_rtol 1.0e-6
-ksp_type gcr
-ksp_view
-log_view
-mat_type aijcusparse
-options_left
-pc_type jacobi
-vec_type cuda
#End of PETSc Option Table entries
There are no unused options.

@Thomas-Ulrich
Copy link
Collaborator Author

Note that I now get with multigrid and no debug:

(base) ulrich@heisenbug:/export/dump/ulrich/section_8_2/scenario-rc$ mpiexec -n 2  static ridge.toml  --output ridgecrest  --mg_strategy twolevel --mg_coarse_level 1  --petsc -ksp_view -ksp_monitor -ksp_converged_reason -ksp_max_it 400 -pc_type mg -mg_levels_ksp_max_it 4 -mg_levels_ksp_type cg -mg_levels_pc_type bjacobi -options_left -ksp_rtol 1.0e-6 -mg_coarse_pc_type gamg -mg_coarse_ksp_type cg -mg_coarse_ksp_rtol 1.0e-1 -mg_coarse_ksp_converged_reason -ksp_type gcr -mg_levels_mat_type aijcusparse -vec_type cuda -mat_type aijcusparse
 

               ___          ___         _____         ___          ___
      ___     /  /\        /__/\       /  /::\       /  /\        /__/\
     /  /\   /  /::\       \  \:\     /  /:/\:\     /  /:/_      |  |::\
    /  /:/  /  /:/\:\       \  \:\   /  /:/  \:\   /  /:/ /\     |  |:|:\
   /  /:/  /  /:/~/::\  _____\__\:\ /__/:/ \__\:| /  /:/ /:/_  __|__|:|\:\
  /  /::\ /__/:/ /:/\:\/__/::::::::\\  \:\ /  /://__/:/ /:/ /\/__/::::| \:\
 /__/:/\:\\  \:\/:/__\/\  \:\~~\~~\/ \  \:\  /:/ \  \:\/:/ /:/\  \:\~~\__\/
 \__\/  \:\\  \::/      \  \:\  ~~~   \  \:\/:/   \  \::/ /:/  \  \:\
      \  \:\\  \:\       \  \:\        \  \::/     \  \:\/:/    \  \:\
       \__\/ \  \:\       \  \:\        \__\/       \  \::/      \  \:\
              \__\/        \__\/                     \__\/        \__\/

                          tandem version 4b6c45e

                       stack size limit = unlimited

                              Worker affinity
    0---------|----------|----------|----------|----------|----------|
    ----------|----------|----------|----------|----------|----------|
    --------8-|----------|----------|----------|----------|----------|
    ----------|----------|----------|----------|----------|----------|
    ----------|------


Ridge:create()
parsing fault_many_trace.csv
Ridge:create()
parsing fault_many_trace.csv
DOFs: 12634590
Mesh size: 36.7959
Multigrid P-levels: 1 2 
[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[1]PETSC ERROR: GPU error
[1]PETSC ERROR: cuSPARSE errorcode 11 (CUSPARSE_STATUS_INSUFFICIENT_RESOURCES) : insufficient resources

@hpc4geo
Copy link
Collaborator

hpc4geo commented Sep 13, 2023 •
edited
Loading

Nice. It is getting closer.

CUSPARSE_STATUS_INSUFFICIENT_RESOURCES refers to all conditions that prevent computing the results. In your case here, I think the error indicates the GPU ran out of memory.

  1. Could you replace -mg_levels_pc_type bjacobi with -mg_levels_pc_type jacobi and re-run?
  2. If the test above also fails, I suggest reducing the memory usage by coarsening the mesh.

I am very puzzled why the log generated contains lots of nan's where there should be run-times. I've never seen that before.

@Thomas-Ulrich
Copy link
Collaborator Author

Thomas-Ulrich commented Sep 14, 2023 •
edited
Loading

  1. did not help
    so that's definitively a out of memory (see also on one GPU clearer error below).

               ___          ___         _____         ___          ___
      ___     /  /\        /__/\       /  /::\       /  /\        /__/\
     /  /\   /  /::\       \  \:\     /  /:/\:\     /  /:/_      |  |::\
    /  /:/  /  /:/\:\       \  \:\   /  /:/  \:\   /  /:/ /\     |  |:|:\
   /  /:/  /  /:/~/::\  _____\__\:\ /__/:/ \__\:| /  /:/ /:/_  __|__|:|\:\
  /  /::\ /__/:/ /:/\:\/__/::::::::\\  \:\ /  /://__/:/ /:/ /\/__/::::| \:\
 /__/:/\:\\  \:\/:/__\/\  \:\~~\~~\/ \  \:\  /:/ \  \:\/:/ /:/\  \:\~~\__\/
 \__\/  \:\\  \::/      \  \:\  ~~~   \  \:\/:/   \  \::/ /:/  \  \:\
      \  \:\\  \:\       \  \:\        \  \::/     \  \:\/:/    \  \:\
       \__\/ \  \:\       \  \:\        \__\/       \  \::/      \  \:\
              \__\/        \__\/                     \__\/        \__\/

                          tandem version 4b6c45e

                       stack size limit = unlimited

                              Worker affinity
    0123456789|0123456789|0123456789|0123456789|0123456789|0123456789|
    0123456789|0123456789|0123456789|0123456789|0123456789|0123456789|
    0123456789|0123456789|0123456789|0123456789|0123456789|0123456789|
    0123456789|0123456789|0123456789|0123456789|0123456789|0123456789|
    0123456789|012345


Ridge:create()
parsing fault_many_trace.csv
DOFs: 12634590
Mesh size: 36.7959
Multigrid P-levels: 1 2 
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: GPU error
[0]PETSC ERROR: cuda error 2 (cudaErrorMemoryAllocation) : out of memory

Can be also see when running nvidia-smi -l,

Thu Sep 14 08:25:19 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.182.03   Driver Version: 470.182.03   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
| 30%   22C    P8     9W / 350W |    567MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:C1:00.0 Off |                  N/A |
| 30%   18C    P8     7W / 350W |      2MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    176938      C   static                            565MiB |
+-----------------------------------------------------------------------------+
Thu Sep 14 08:25:24 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.182.03   Driver Version: 470.182.03   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
| 30%   31C    P2   118W / 350W |  22205MiB / 24268MiB |     69%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:C1:00.0 Off |                  N/A |
| 30%   18C    P8     7W / 350W |      2MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

Indeed when reducing the mesh size to 3914 nodes 25174 elements (h_fault=1)
(DOFs: 608700), I can run the setup on the 2 GPUs.

@Thomas-Ulrich
Copy link
Collaborator Author

Thomas-Ulrich commented Sep 14, 2023 •
edited
Loading

Overall, the GPU code converges much slowly to solution.
Here is the log running on CPUs as mpiexec -n 2 static ridge.toml --output ridgecrest --mg_strategy twolevel --mg_coarse_level 1 --petsc -ksp_view -ksp_monitor -ksp_converged_reason -ksp_max_it 400 -pc_type mg -mg_levels_ksp_max_it 4 -mg_levels_ksp_type cg -mg_levels_pc_type bjacobi -options_left -ksp_rtol 1.0e-6 -mg_coarse_pc_type gamg -mg_coarse_ksp_type cg -mg_coarse_ksp_rtol 1.0e-1 -mg_coarse_ksp_converged_reason -ksp_type gcr

  0 KSP Residual norm 1.734965842704e+04 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 18
  1 KSP Residual norm 1.031526607053e+02 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 21
  2 KSP Residual norm 3.572058785172e+01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 15
  3 KSP Residual norm 1.140666097752e+01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 18
  4 KSP Residual norm 4.219243222064e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 16
  5 KSP Residual norm 1.726757190108e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 19
  6 KSP Residual norm 7.684999488409e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 16
  7 KSP Residual norm 3.047540104055e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 16
  8 KSP Residual norm 1.073314964141e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 16
  9 KSP Residual norm 3.571480635536e-02 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 15
 10 KSP Residual norm 1.270504655145e-02 
Linear solve converged due to CONVERGED_RTOL iterations 10

The same setup converges much more slowly when running with the additional options:
-mg_levels_mat_type aijcusparse -vec_type cuda -mat_type aijcusparse

  0 KSP Residual norm 1.734965842704e+04 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  1 KSP Residual norm 1.112811862053e+02 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  2 KSP Residual norm 4.532554406057e+01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  3 KSP Residual norm 2.375197340723e+01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  4 KSP Residual norm 1.672539760740e+01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  5 KSP Residual norm 1.414756771772e+01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  6 KSP Residual norm 1.121542586970e+01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  7 KSP Residual norm 8.941253504767e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  8 KSP Residual norm 7.158088359090e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
  9 KSP Residual norm 5.939360463184e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 10 KSP Residual norm 5.106714061854e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 11 KSP Residual norm 4.616036762828e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 12 KSP Residual norm 4.151022296180e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 13 KSP Residual norm 3.745663945543e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 14 KSP Residual norm 3.342640042622e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 15 KSP Residual norm 2.870591058752e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 16 KSP Residual norm 2.577647604225e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 17 KSP Residual norm 2.329289494495e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 18 KSP Residual norm 1.958163659908e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 19 KSP Residual norm 1.648044186263e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 20 KSP Residual norm 1.417638055974e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 21 KSP Residual norm 1.280970685123e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 22 KSP Residual norm 1.101320821088e+00 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 23 KSP Residual norm 9.074620117579e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 24 KSP Residual norm 7.159338239786e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 25 KSP Residual norm 6.068813693601e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 26 KSP Residual norm 5.600414430325e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 27 KSP Residual norm 5.085937212294e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 28 KSP Residual norm 4.576119448521e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 29 KSP Residual norm 3.928048766129e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 30 KSP Residual norm 3.234931639246e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 31 KSP Residual norm 2.947903349793e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 32 KSP Residual norm 2.695541476443e-01 
    Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
 33 KSP Residual norm 2.434954480914e-01 
 (...)
     Linear mg_coarse_ solve converged due to CONVERGED_RTOL iterations 37
103 KSP Residual norm 1.668932459434e-02

Note also

There is one unused database option. It is:
Option left: name:-mg_levels_mat_type value: aijcusparse

@Thomas-Ulrich
Copy link
Collaborator Author

Thomas-Ulrich commented Sep 15, 2023 •
edited
Loading

(for tracking the cause of the difference in the residual decrease)
Here are the two logs, on 2 nodes:
tandem_CPU.log
tandem_GPU.log

@Thomas-Ulrich
Copy link
Collaborator Author

Thomas-Ulrich commented Sep 15, 2023 •
edited
Loading

Testing the matrix free I get:

base) ulrich@heisenbug:/export/dump/ulrich/section_8_2/scenario-rc$ mpiexec -n 2 static ridge.toml  --output ridgecrest  --matrix_free --mg_strategy twolevel --mg_coarse_level 1 --petsc -ksp_view -ksp_monitor -ksp_converged_reason -ksp_max_it 40 -pc_type mg -mg_levels_ksp_max_it 4 -mg_levels_ksp_type cg -mg_levels_pc_type bjacobi -options_left -ksp_rtol 1.0e-6 -mg_coarse_pc_type gamg -mg_coarse_ksp_type cg -mg_coarse_ksp_rtol 1.0e-1 -mg_coarse_ksp_converged_reason -ksp_type gcr  -vec_type cuda -mat_type aijcusparse

optional argument in parameter sequence
Usage: static [options] config 

Positional arguments:
config                  Configuration file (.toml)

Optional arguments:
-h --help               shows help message and exits
-v --version            prints version information and exits
--petsc                 PETSc options, must be passed last!
--resolution            Non-negative resolution parameter
--method           
--type             
--lib              
--scenario         
--matrix_free      
--test_matrix_free 
--mg_coarse_level  
--mg_strategy      
--profile               Run static in profile mode. The parameter controls the amount of repetitions.
--output                Output file name
--mesh_file        
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------

@hpc4geo
Copy link
Collaborator

hpc4geo commented Sep 15, 2023 •
edited
Loading

Thanks for the log files. I will take a look. To use matrix-free you need --matrix_free true. However don't waste your time with this option. Despite what you might think it will NOT reduce the memory foot print. When I was referring to 'matrix-free' and reducing the memory usage I was explicitly making reference to branch dmay/solver-dev . This branch is not yet GPU compatible.

@hpc4geo
Copy link
Collaborator

hpc4geo commented Sep 15, 2023

The name of the issue is too vague. The current issues are not related to getting something running on a GPU. Closing this and make a new issue.

@hpc4geo hpc4geo closed this as completed Sep 15, 2023
@hpc4geo hpc4geo mentioned this issue Sep 15, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Thomas-Ulrich @hpc4geo