The problem of building wasp

[ljeft]

Hello
I got an error running "./update_and_rebuild_wasp.sh "following the install moose tutorial

./update_and_rebuild_wasp.sh
CMake Error: The source directory "/home/sohar/projects/moose/framework/contrib/wasp" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.
Error: configure step for WASP failed to complete successfully`

This is my diagnostics script

Thu Feb 29 18:48:42 CST 2024
No LSB modules are available.

System Arch: Distributor ID: Ubuntu Description: Ubuntu 22.04.3 LTS Release: 22.04 Codename: jammy

MOOSE Package Version: Custom Build

CPU Count: 28

Memory Free: 7910.754 MB

Variable `which $CC` check:
/usr/bin/mpicc

$CC --version:
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

MPICC:
which mpicc:
        /usr/bin/mpicc
mpicc -show:
        gcc -I/usr/lib/x86_64-linux-gnu/openmpi/include -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi

COMPILER gcc:
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Python: NO PYTHON DETECTED

MODULES NOT AVAILABLE

PETSC_DIR not set

ENVIRONMENT:
CC=mpicc
CXX=mpicxx
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
DISPLAY=:0
F77=mpif77
F90=mpif90
FC=mpif90
HOME=/home/sohar
HOSTTYPE=x86_64
LANG=C.UTF-8
LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64:
LESSCLOSE=/usr/bin/lesspipe %s %s
LESSOPEN=| /usr/bin/lesspipe %s
LOGNAME=sohar
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
METHODS=opt
MOOSE_JOBS=6
NAME=DESKTOP-RPC0OMG
OLDPWD=/home/sohar/projects/moose
PATH=/usr/local/cuda-11.8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files/WindowsApps/CanonicalGroupLimited.Ubuntu22.04LTS_2204.3.63.0_x64__79rhkp1fndgsc:/mnt/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/bin:/mnt/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/libnvvp:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0/:/mnt/c/Windows/System32/OpenSSH/:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/Program Files/NVIDIA Corporation/NVIDIA NvDLISR:/mnt/c/Program Files/Microsoft SQL Server/150/Tools/Binn/:/mnt/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn/:/mnt/e/Windows Kits/10/Windows Performance Toolkit/:/mnt/c/Program Files/NVIDIA Corporation/Nsight Compute 2022.3.0/:/mnt/e/Git/cmd:/mnt/c/Users/DZKD/AppData/Local/Microsoft/WindowsApps:/mnt/d/Users/DZKD/AppData/Local/Programs/Microsoft VS Code/bin:/snap/bin
PULSE_SERVER=unix:/mnt/wslg/PulseServer
PWD=/home/sohar/projects/moose/scripts
SHELL=/bin/bash
SHLVL=1
TERM=xterm-256color
USER=sohar
WAYLAND_DISPLAY=wayland-0
WSL2_GUI_APPS_ENABLED=1
WSLENV=
WSL_DISTRO_NAME=Ubuntu-22.04
WSL_INTEROP=/run/WSL/348_interop
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
XDG_RUNTIME_DIR=/run/user/1000/
_=/usr/bin/env

[milljm]

Interesting... the update_and_rebuild_wasp.sh script should have pulled in all the necessary submodules. But it looks like it didn't for you, for some reason or another.

I've run into this error when building WASP as a Conda package. The solution is to recursively pull all of WASP's submodules:

cd moose/framework/contrib/wasp
git submodule update --init --recursive

... and then run the update_and_rebuild_wasp.sh script.

[ljeft]

I am going to try to use conda to install moose. I want to use GPU but I don't know which step to make PETSc support cuda

[milljm]

@grmnptr I think there will be minimal changes required in order to use our Conda supplied packages along with Cuda support. I could be wrong. But I am actively creating Conda moose packages to support Cuda.

[ljeft]

According to HPC download method, when I run update_and_rebuild_pets.sh script, I can modify configure_petsc.sh script to add with-cuda=1 to make PETSc support cuda but I don't know how to do other methods.

[milljm]

hmmm if that is all it takes to add Cuda support to PETSc, then yes sir! That should do the trick:

./update_and_rebuild_petsc.sh --with-cuda=1

If you do this, and you are still wanting to use Conda packages, you should probably refrain from install the full gambit (don't install moose-dev). Instead, install what only is needed to build PETSc: conda create -n compilers_only moose-mpich.

I am going to try to use conda to install moose

But even this is not recommended if you are operating on an HPC cluster. Your cluster will most certainly already have a much-better MPI wrapper, than what we provide with moose-mpich.

[ljeft]

Having previously installed moose on hpc and failed to get PETSc to support CUDA, I am now trying to get moose to call GPU on pc. I was hoping to get PETSc to support CUDA when moose was installed, but following the HPC approach didn't work. I am now wondering how to use other moose installation methods and be able to rebuild PETSc.

[ljeft]

Unfortunately, when I downloaded moose using the conda method, there was an error at run ```
cd ~/projects/moose/test
make -j 6
This is the error

Compiling C++ (in opt mode) /home/sohar/projects/moose/framework/build/unity_src/base_Unity.C...
Compiling C++ (in opt mode) /home/sohar/projects/moose/framework/build/unity_src/interfaces_Unity.C...
Compiling C++ (in opt mode) /home/sohar/projects/moose/framework/build/unity_src/geomsearch_Unity.C...
Compiling C++ (in opt mode) /home/sohar/projects/moose/framework/build/unity_src/bcs_Unity.C...
Compiling C++ (in opt mode) /home/sohar/projects/moose/framework/build/unity_src/distributions_Unity.C...
Compiling C++ (in opt mode) /home/sohar/projects/moose/framework/build/unity_src/libtorch_Unity.C...
Compiling C++ (in opt mode) /home/sohar/projects/moose/framework/build/unity_src/postprocessors_Unity.C...
Compiling C++ (in opt mode) /home/sohar/projects/moose/framework/build/unity_src/bounds_Unity.C...
In file included from /home/sohar/projects/moose/framework/build/unity_src/base_Unity.C:19:
/home/sohar/projects/moose/framework/src/base/MooseServer.C: In member function 'bool MooseServer::addParametersToList(wasp::DataArray&, const InputParameters&, const std::set<std::__cxx11::basic_string<char> >&, int, int, int, int, const string&)':
/home/sohar/projects/moose/framework/src/base/MooseServer.C:529:9: error: 'client_snippet_support' was not declared in this scope
  529 |     if (client_snippet_support && !default_value.empty())
      |         ^~~~~~~~~~~~~~~~~~~~~~
/home/sohar/projects/moose/framework/src/base/MooseServer.C:531:32: error: 'm_text_format_snippet' is not a member of 'wasp::lsp'
  531 |       text_format = wasp::lsp::m_text_format_snippet;
      |                                ^~~~~~~~~~~~~~~~~~~~~
/home/sohar/projects/moose/framework/src/base/MooseServer.C:536:32: error: 'm_text_format_plaintext' is not a member of 'wasp::lsp'
  536 |       text_format = wasp::lsp::m_text_format_plaintext;
      |                                ^~~~~~~~~~~~~~~~~~~~~~~
/home/sohar/projects/moose/framework/src/base/MooseServer.C:557:57: error: too many arguments to function 'bool wasp::lsp::buildCompletionObject(wasp::DataObject&, std::ostream&, const string&, int, int, int, int, const string&, int, const string&, const string&, bool, bool)'
  557 |                                              text_format);
      |                                                         ^
In file included from /home/sohar/projects/moose/framework/build/header_symlinks/MooseServer.h:13,
                 from /home/sohar/projects/moose/framework/src/base/MooseApp.C:53,
                 from /home/sohar/projects/moose/framework/build/unity_src/base_Unity.C:12:
/home/sohar/miniforge/envs/moose/wasp/include/wasplsp/LSP.h:604:6: note: declared here
  604 | bool buildCompletionObject( DataObject        & object          ,
      |      ^~~~~~~~~~~~~~~~~~~~~
In file included from /home/sohar/projects/moose/framework/build/unity_src/base_Unity.C:19:
/home/sohar/projects/moose/framework/src/base/MooseServer.C: In member function 'bool MooseServer::addSubblocksToList(wasp::DataArray&, const string&, int, int, int, int, const string&, bool)':
/home/sohar/projects/moose/framework/src/base/MooseServer.C:618:25: error: 'client_snippet_support' was not declared in this scope
  618 |       int text_format = client_snippet_support ? wasp::lsp::m_text_format_snippet
      |                         ^~~~~~~~~~~~~~~~~~~~~~
/home/sohar/projects/moose/framework/src/base/MooseServer.C:618:61: error: 'm_text_format_snippet' is not a member of 'wasp::lsp'
  618 |       int text_format = client_snippet_support ? wasp::lsp::m_text_format_snippet
      |                                                             ^~~~~~~~~~~~~~~~~~~~~
/home/sohar/projects/moose/framework/src/base/MooseServer.C:619:61: error: 'm_text_format_plaintext' is not a member of 'wasp::lsp'
  619 |                                                : wasp::lsp::m_text_format_plaintext;
      |                                                             ^~~~~~~~~~~~~~~~~~~~~~~
/home/sohar/projects/moose/framework/src/base/MooseServer.C:654:59: error: too many arguments to function 'bool wasp::lsp::buildCompletionObject(wasp::DataObject&, std::ostream&, const string&, int, int, int, int, const string&, int, const string&, const string&, bool, bool)'
  654 |                                                text_format);
      |                                                           ^
In file included from /home/sohar/projects/moose/framework/build/header_symlinks/MooseServer.h:13,
                 from /home/sohar/projects/moose/framework/src/base/MooseApp.C:53,
                 from /home/sohar/projects/moose/framework/build/unity_src/base_Unity.C:12:
/home/sohar/miniforge/envs/moose/wasp/include/wasplsp/LSP.h:604:6: note: declared here
  604 | bool buildCompletionObject( DataObject        & object          ,
      |      ^~~~~~~~~~~~~~~~~~~~~
In file included from /home/sohar/projects/moose/framework/build/unity_src/base_Unity.C:19:
/home/sohar/projects/moose/framework/src/base/MooseServer.C: In member function 'bool MooseServer::addValuesToList(wasp::DataArray&, const InputParameters&, const std::set<std::__cxx11::basic_string<char> >&, const string&, const std::set<std::__cxx11::basic_string<char> >&, int, int, int, int)':
/home/sohar/projects/moose/framework/src/base/MooseServer.C:791:9: error: 'client_snippet_support' was not declared in this scope
  791 |     if (client_snippet_support)
      |         ^~~~~~~~~~~~~~~~~~~~~~
/home/sohar/projects/moose/framework/src/base/MooseServer.C:793:32: error: 'm_text_format_snippet' is not a member of 'wasp::lsp'
  793 |       text_format = wasp::lsp::m_text_format_snippet;
      |                                ^~~~~~~~~~~~~~~~~~~~~
/home/sohar/projects/moose/framework/src/base/MooseServer.C:798:32: error: 'm_text_format_plaintext' is not a member of 'wasp::lsp'
  798 |       text_format = wasp::lsp::m_text_format_plaintext;
      |                                ^~~~~~~~~~~~~~~~~~~~~~~
/home/sohar/projects/moose/framework/src/base/MooseServer.C:818:57: error: too many arguments to function 'bool wasp::lsp::buildCompletionObject(wasp::DataObject&, std::ostream&, const string&, int, int, int, int, const string&, int, const string&, const string&, bool, bool)'
  818 |                                              text_format);
      |                                                         ^
In file included from /home/sohar/projects/moose/framework/build/header_symlinks/MooseServer.h:13,
                 from /home/sohar/projects/moose/framework/src/base/MooseApp.C:53,
                 from /home/sohar/projects/moose/framework/build/unity_src/base_Unity.C:12:
/home/sohar/miniforge/envs/moose/wasp/include/wasplsp/LSP.h:604:6: note: declared here
  604 | bool buildCompletionObject( DataObject        & object          ,
      |      ^~~~~~~~~~~~~~~~~~~~~
Compiling C++ (in opt mode) /home/sohar/projects/moose/framework/build/unity_src/fvbcs_Unity.C...
make: *** [/home/sohar/projects/moose/framework/build.mk:150: /home/sohar/projects/moose/framework/build/unity_src/base_Unity.x86_64-conda-linux-gnu.opt.lo] Error 1
make: *** Waiting for unfinished jobs....

This is the script diagnostic

Sun Mar  3 13:46:22 CST 2024
No LSB modules are available.

System Arch: Distributor ID: Ubuntu Description: Ubuntu 22.04.4 LTS Release: 22.04 Codename: jammy

MOOSE Package Version: Custom Build

CPU Count: 28

Memory Free: 12585.598 MB

Variable `which $CC` check:
/home/sohar/miniforge/envs/moose/bin/mpicc

$CC --version:
x86_64-conda-linux-gnu-cc (conda-forge gcc 10.4.0-19) 10.4.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

MPICC:
which mpicc:
        /home/sohar/miniforge/envs/moose/bin/mpicc
mpicc -show:
        x86_64-conda-linux-gnu-cc -I/home/sohar/miniforge/envs/moose/include -I/home/sohar/miniforge/envs/moose/include -L/home/sohar/miniforge/envs/moose/lib -Wl,-rpath,/home/sohar/miniforge/envs/moose/lib -I/home/sohar/miniforge/envs/moose/include -L/home/sohar/miniforge/envs/moose/lib -Wl,-rpath -Wl,/home/sohar/miniforge/envs/moose/lib -Wl,--enable-new-dtags -lmpi

COMPILER x86_64-conda-linux-gnu-cc:
x86_64-conda-linux-gnu-cc (conda-forge gcc 10.4.0-19) 10.4.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Python:
        /home/sohar/miniforge/envs/moose/bin/python
        Python 3.10.8

MODULES NOT AVAILABLE

PETSc configure:
static const char *petscconfigureoptions = "--with-64-bit-indices --with-cxx-dialect=C++17 --with-debugging=no --with-fortran-bindings=0 --with-mpi=1 --with-openmp=1 --with-shared-libraries=1 --with-sowing=0 --download-fblaslapack=1 --download-hypre=1 --download-metis=1 --download-mumps=1 --download-ptscotch=1 --download-parmetis=1 --download-scalapack=1 --download-slepc=1 --download-strumpack=1 --download-superlu_dist=1 --with-hdf5-dir=${PREFIX} --with-make-np=16 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-x=0 --with-ssl=0 --with-mpi-dir=/home/sohar/miniforge/envs/moose AR=${PREFIX}/bin/x86_64-conda-linux-gnu-ar RANLIB=${PREFIX}/bin/x86_64-conda-linux-gnu-ranlib CFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/sohar/miniforge/envs/moose/include   -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/sohar/miniforge/envs/moose/include   -march=nocona -mtune=haswell" CXXFLAGS="-fvisibility-inlines-hidden -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem ${PREFIX}/include  -march=nocona -mtune=haswell" CPPFLAGS="-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/sohar/miniforge/envs/moose/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/sohar/miniforge/envs/moose/include" FFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/sohar/miniforge/envs/moose/include   -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/sohar/miniforge/envs/moose/include   -I/home/sohar/miniforge/envs/moose/include" FCFLAGS="-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/sohar/miniforge/envs/moose/include   -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/sohar/miniforge/envs/moose/include   -I/home/sohar/miniforge/envs/moose/include" LDFLAGS="-Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/home/sohar/miniforge/envs/moose/lib -Wl,-rpath-link,/home/sohar/miniforge/envs/moose/lib -L/home/sohar/miniforge/envs/moose/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/home/sohar/miniforge/envs/moose/lib -Wl,-rpath-link,/home/sohar/miniforge/envs/moose/lib -L/home/sohar/miniforge/envs/moose/lib" --prefix=/home/sohar/miniforge/envs/moose";

PETSc linked libraries:
        linux-vdso.so.1 (0x00007ffda2ef5000)
        libHYPRE-2.29.0.so => /home/sohar/miniforge/envs/moose/lib/./libHYPRE-2.29.0.so (0x00007f7d0adc4000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7d0adb4000)
        libstrumpack.so.7.1 => /home/sohar/miniforge/envs/moose/lib/./libstrumpack.so.7.1 (0x00007f7d09d95000)
        libscalapack.so.2.2 => /home/sohar/miniforge/envs/moose/lib/./libscalapack.so.2.2 (0x00007f7d0969e000)
        libsuperlu_dist.so.8 => /home/sohar/miniforge/envs/moose/lib/./libsuperlu_dist.so.8 (0x00007f7d0951a000)
        libparmetis.so => /home/sohar/miniforge/envs/moose/lib/./libparmetis.so (0x00007f7d094da000)
        libmetis.so => /home/sohar/miniforge/envs/moose/lib/./libmetis.so (0x00007f7d0946f000)
        libhdf5.so.200 => /home/sohar/miniforge/envs/moose/lib/./libhdf5.so.200 (0x00007f7d0903d000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7d08f56000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7d08f51000)
        libmpifort.so.12 => /home/sohar/miniforge/envs/moose/lib/./libmpifort.so.12 (0x00007f7d08f00000)
        libmpi.so.12 => /home/sohar/miniforge/envs/moose/lib/./libmpi.so.12 (0x00007f7d08ae4000)
        libgfortran.so.5 => /home/sohar/miniforge/envs/moose/lib/./libgfortran.so.5 (0x00007f7d08939000)
        libgcc_s.so.1 => /home/sohar/miniforge/envs/moose/lib/./libgcc_s.so.1 (0x00007f7d0891e000)
        libstdc++.so.6 => /home/sohar/miniforge/envs/moose/lib/./libstdc++.so.6 (0x00007f7d0873b000)
        libgomp.so.1 => /home/sohar/miniforge/envs/moose/lib/./libgomp.so.1 (0x00007f7d086ff000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7d084d6000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7d0cca5000)
        libquadmath.so.0 => /home/sohar/miniforge/envs/moose/lib/././libquadmath.so.0 (0x00007f7d0849b000)
        libzfp.so.0 => /home/sohar/miniforge/envs/moose/lib/././libzfp.so.0 (0x00007f7d08447000)
        libmpicxx.so.12 => /home/sohar/miniforge/envs/moose/lib/././libmpicxx.so.12 (0x00007f7d08428000)
        libcrypto.so.1.1 => /home/sohar/miniforge/envs/moose/lib/././libcrypto.so.1.1 (0x00007f7d08156000)
        libcurl.so.4 => /home/sohar/miniforge/envs/moose/lib/././libcurl.so.4 (0x00007f7d080b3000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7d080ac000)
        libz.so.1 => /home/sohar/miniforge/envs/moose/lib/././libz.so.1 (0x00007f7d08091000)
        libnghttp2.so.14 => /home/sohar/miniforge/envs/moose/lib/./././libnghttp2.so.14 (0x00007f7d08065000)
        libssh2.so.1 => /home/sohar/miniforge/envs/moose/lib/./././libssh2.so.1 (0x00007f7d08021000)
        libssl.so.1.1 => /home/sohar/miniforge/envs/moose/lib/./././libssl.so.1.1 (0x00007f7d07f8f000)
        libgssapi_krb5.so.2 => /home/sohar/miniforge/envs/moose/lib/./././libgssapi_krb5.so.2 (0x00007f7d07f3c000)
        libkrb5.so.3 => /home/sohar/miniforge/envs/moose/lib/././././libkrb5.so.3 (0x00007f7d07e65000)
        libk5crypto.so.3 => /home/sohar/miniforge/envs/moose/lib/././././libk5crypto.so.3 (0x00007f7d07e4b000)
        libcom_err.so.3 => /home/sohar/miniforge/envs/moose/lib/././././libcom_err.so.3 (0x00007f7d07e45000)
        libkrb5support.so.0 => /home/sohar/miniforge/envs/moose/lib/././././libkrb5support.so.0 (0x00007f7d07e36000)
        libkeyutils.so.1 => /home/sohar/miniforge/envs/moose/lib/././././libkeyutils.so.1 (0x00007f7d07e2d000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f7d07e19000)

ENVIRONMENT:
ADDR2LINE=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-addr2line
AR=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-ar
AS=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-as
BUILD=x86_64-conda-linux-gnu
CC=mpicc
CC_FOR_BUILD=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-cc
CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/sohar/miniforge/envs/moose/include
CMAKE_ARGS=-DCMAKE_AR=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-ar -DCMAKE_CXX_COMPILER_AR=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_C_COMPILER_AR=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_RANLIB=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-ranlib -DCMAKE_CXX_COMPILER_RANLIB=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_C_COMPILER_RANLIB=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_LINKER=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-ld -DCMAKE_STRIP=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-strip
CMAKE_PREFIX_PATH=/home/sohar/miniforge/envs/moose:/home/sohar/miniforge/envs/moose/x86_64-conda-linux-gnu/sysroot/usr
CONDA_BUILD_SYSROOT=/home/sohar/miniforge/envs/moose/x86_64-conda-linux-gnu/sysroot
CONDA_DEFAULT_ENV=moose
CONDA_EXE=/home/sohar/miniforge/bin/conda
CONDA_PREFIX=/home/sohar/miniforge/envs/moose
CONDA_PREFIX_1=/home/sohar/miniforge
CONDA_PROMPT_MODIFIER=(moose)
CONDA_PYTHON_EXE=/home/sohar/miniforge/bin/python
CONDA_SHLVL=2
CONDA_TOOLCHAIN_BUILD=x86_64-conda-linux-gnu
CONDA_TOOLCHAIN_HOST=x86_64-conda-linux-gnu
CPP=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-cpp
CPPFLAGS=-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/sohar/miniforge/envs/moose/include
CXX=mpicxx
CXXFILT=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-c++filt
CXXFLAGS=-fvisibility-inlines-hidden  -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/sohar/miniforge/envs/moose/include -std=c++17
CXX_FOR_BUILD=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-c++
C_INCLUDE_PATH=/home/sohar/miniforge/envs/moose/include
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
DEBUG_CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -ffunction-sections -pipe -isystem /home/sohar/miniforge/envs/moose/include
DEBUG_CPPFLAGS=-D_DEBUG -D_FORTIFY_SOURCE=2 -Og -isystem /home/sohar/miniforge/envs/moose/include
DEBUG_CXXFLAGS=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -ffunction-sections -pipe -isystem /home/sohar/miniforge/envs/moose/include
DEBUG_FFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/sohar/miniforge/envs/moose/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fcheck=all -fbacktrace -fimplicit-none -fvar-tracking-assignments -ffunction-sections -pipe
DEBUG_FORTRANFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/sohar/miniforge/envs/moose/include -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fcheck=all -fbacktrace -fimplicit-none -fvar-tracking-assignments -ffunction-sections -pipe
DISPLAY=192.168.0.1:0.0
ELFEDIT=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-elfedit
Eigen3_DIR=/home/sohar/miniforge/envs/moose/libmesh/include/Eigen
F77=mpif77
F90=mpif90
F95=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-f95
FC=mpif90
FC_FOR_BUILD=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-gfortran
FFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/sohar/miniforge/envs/moose/include
FORTRANFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/sohar/miniforge/envs/moose/include
GCC=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-gcc
GCC_AR=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-gcc-ar
GCC_NM=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-gcc-nm
GCC_RANLIB=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-gcc-ranlib
GFORTRAN=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-gfortran
GPROF=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-gprof
GSETTINGS_SCHEMA_DIR=/home/sohar/miniforge/envs/moose/share/glib-2.0/schemas
GSETTINGS_SCHEMA_DIR_CONDA_BACKUP=
GXX=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-g++
HDF5_DIR=/home/sohar/miniforge/envs/moose
HOME=/home/sohar
HOST=x86_64-conda-linux-gnu
HOSTTYPE=x86_64
LANG=C.UTF-8
LD=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-ld
LDFLAGS=-Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/home/sohar/miniforge/envs/moose/lib -Wl,-rpath-link,/home/sohar/miniforge/envs/moose/lib -L/home/sohar/miniforge/envs/moose/lib
LD_GOLD=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-ld.gold
LESSCLOSE=/usr/bin/lesspipe %s %s
LESSOPEN=| /usr/bin/lesspipe %s
LIBMESH_DIR=/home/sohar/miniforge/envs/moose/libmesh
LOGNAME=sohar
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
MESON_ARGS=--buildtype release
MOOSE_NO_CODESIGN=true
MOTD_SHOWN=update-motd
MPIHOME=/home/sohar/miniforge/envs/moose
NAME=DESKTOP-RPC0OMG
NM=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-nm
OBJCOPY=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-objcopy
OBJDUMP=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-objdump
OLDPWD=/home/sohar/projects/moose/test
PATH=/home/sohar/miniforge/envs/moose/bin:/home/sohar/miniforge/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files/WindowsApps/CanonicalGroupLimited.Ubuntu22.04LTS_2204.3.63.0_x64__79rhkp1fndgsc:/mnt/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/bin:/mnt/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/libnvvp:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/Program Files/NVIDIA Corporation/NVIDIA NvDLISR:/mnt/c/Program Files/Microsoft SQL Server/150/Tools/Binn:/mnt/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn:/mnt/e/Windows Kits/10/Windows Performance Toolkit:/mnt/c/Program Files/NVIDIA Corporation/Nsight Compute 2022.3.0:/mnt/e/Git/cmd:/mnt/c/Users/DZKD/AppData/Local/Microsoft/WindowsApps:/mnt/d/Users/DZKD/AppData/Local/Programs/Microsoft VS Code/bin:/snap/bin:/home/sohar/miniforge/envs/moose/wasp/bin
PETSC_DIR=/home/sohar/miniforge/envs/moose
PROJ_LIB=/home/sohar/miniforge/envs/moose/share/proj
PROJ_NETWORK=ON
PULSE_SERVER=unix:/mnt/wslg/PulseServer
PWD=/home/sohar/projects/moose/scripts
RANLIB=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-ranlib
READELF=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-readelf
SHELL=/bin/bash
SHLVL=1
SIZE=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-size
STRINGS=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-strings
STRIP=/home/sohar/miniforge/envs/moose/bin/x86_64-conda-linux-gnu-strip
TERM=xterm-256color
USER=sohar
VTKINCLUDE_DIR=/home/sohar/miniforge/envs/moose/libmesh-vtk/include/vtk-9.2
VTKLIB_DIR=/home/sohar/miniforge/envs/moose/libmesh-vtk/lib
WASP_DIR=/home/sohar/miniforge/envs/moose/wasp
WAYLAND_DISPLAY=wayland-0
WSL2_GUI_APPS_ENABLED=1
WSLENV=
WSL_DISTRO_NAME=Ubuntu-22.04
WSL_INTEROP=/run/WSL/17100_interop
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
XDG_RUNTIME_DIR=/run/user/1000/
XML_CATALOG_FILES=file:///home/sohar/miniforge/envs/moose/etc/xml/catalog file:///etc/xml/catalog
_=/usr/bin/env
_CE_CONDA=
_CE_M=
_CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_cos6_linux_gnu
build_alias=x86_64-conda-linux-gnu
host_alias=x86_64-conda-linux-gnu

[ljeft]

I type pwd to show that there are 4 folders, using the use of 4 ".. / "is still wrong

(moose) sohar@DESKTOP-RPC0OMG:~/projects/fish$ pwd
/home/sohar/projects/fish
(moose) sohar@DESKTOP-RPC0OMG:~/projects/fish$ sudo mpirun -n 8  --allow-run-as-root  ../../../../fish-opt -i ./ad_lid_d
riven_action_stabilized_steady.i
--------------------------------------------------------------------------
mpirun was unable to launch the specified application as it could not access
or execute an executable:

Executable: ../../../../fish-opt
Node: DESKTOP-RPC0OMG

while attempting to start process rank 0.
--------------------------------------------------------------------------
8 total processes failed to start

[milljm]

Using sudo while in a necessary environment (moose), will not work. Sudo will not pass/inherit your non-elevated permissions.

Is there a reason you need to use sudo at all here? It would seem you are operating within your home directory, negating the need all together.

Just FYI, to see what I mean about the environment differences:

conda activate moose
sudo env

and then

env

[ljeft]

I added sudo because I didn't use "sudo" to inform me of permission denied before

(moose) sohar@DESKTOP-RPC0OMG:~/projects/fish$ mpirun -n 8 ../fish -i ./ad_lid_driven_action_stabilized_steady.i
[proxy:0:0@DESKTOP-RPC0OMG] HYDU_create_process (utils/launch/launch.c:73): execvp error on file ../fish (Permission denied)
[proxy:0:0@DESKTOP-RPC0OMG] HYDU_create_process (utils/launch/launch.c:73): execvp error on file ../fish (Permission denied)
[proxy:0:0@DESKTOP-RPC0OMG] HYDU_create_process (utils/launch/launch.c:73): execvp error on file ../fish (Permission denied)
[proxy:0:0@DESKTOP-RPC0OMG] HYDU_create_process (utils/launch/launch.c:73): execvp error on file ../fish (Permission denied)
[proxy:0:0@DESKTOP-RPC0OMG] HYDU_create_process (utils/launch/launch.c:73): execvp error on file ../fish (Permission denied)
[proxy:0:0@DESKTOP-RPC0OMG] HYDU_create_process (utils/launch/launch.c:73): execvp error on file ../fish (Permission denied)
[proxy:0:0@DESKTOP-RPC0OMG] HYDU_create_process (utils/launch/launch.c:73): execvp error on file ../fish (Permission denied)
[proxy:0:0@DESKTOP-RPC0OMG] HYDU_create_process (utils/launch/launch.c:73): execvp error on file ../fish (Permission denied)
(moose) sohar@DESKTOP-RPC0OMG:~/projects/fish$ sudo mpirun -n 8 ../fish -i ./ad_lid_driven_action_stabilized_steady.i
[sudo] password for sohar:
--------------------------------------------------------------------------
mpirun has detected an attempt to run as root.

Running as root is *strongly* discouraged as any mistake (e.g., in
defining TMPDIR) or bug can result in catastrophic damage to the OS
file system, leaving your system in an unusable state.

We strongly suggest that you run mpirun as a non-root user.

You can override this protection by adding the --allow-run-as-root option
to the cmd line or by setting two environment variables in the following way:
the variable OMPI_ALLOW_RUN_AS_ROOT=1 to indicate the desire to override this
protection, and OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 to confirm the choice and
add one more layer of certainty that you want to do so.
We reiterate our advice against doing so - please proceed at your own risk.
--------------------------------------------------------------------------
(moose) sohar@DESKTOP-RPC0OMG:~/projects/fish$ mpirun -n 8 --allow-run-as-root  ../fish -i ./ad_lid_driven_action_stabilized_steady.i
[mpiexec@DESKTOP-RPC0OMG] match_arg (utils/args/args.c:163): unrecognized argument allow-run-as-root
[mpiexec@DESKTOP-RPC0OMG] HYDU_parse_array (utils/args/args.c:178): argument matching returned error
[mpiexec@DESKTOP-RPC0OMG] parse_args (ui/mpich/utils.c:1639): error parsing input array
[mpiexec@DESKTOP-RPC0OMG] HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1691): unable to parse user arguments
[mpiexec@DESKTOP-RPC0OMG] main (ui/mpich/mpiexec.c:127): error parsing parameters

[GiudGiud]

../fish does not exist of course you are getting an error.

This is ultra basic linux / using the terminal. You need to be able to navigate folders to use moose.
when you type a path to an executable or to a file, you need to understand what it means. please follow a command line tutorial to get up to speed on this

[ljeft]

Yes, my question was stupid, I have modified it and successfully run the example.

(moose) sohar@DESKTOP-RPC0OMG:~/projects/fish/example$ mpiexec -n 6  ../fish-opt -i ./ad_lid_driven_action_stabilized_st
eady.i
Framework Information:
MOOSE Version:           git commit 778cc52c3e on 2024-03-02
LibMesh Version:
PETSc Version:           3.20.3
SLEPc Version:           3.20.1
Current Time:            Wed Mar  6 01:20:48 2024
Executable Timestamp:    Tue Mar  5 15:23:34 2024

Parallelism:
  Num Processors:          6
  Num Threads:             1

Mesh:
  Parallel Type:           replicated
  Mesh Dimension:          2
  Spatial Dimension:       2
  Nodes:
    Total:                 289
    Local:                 57
    Min/Max/Avg:           39/58/48
  Elems:
    Total:                 256
    Local:                 42
    Min/Max/Avg:           42/43/42
  Num Subdomains:          1
  Num Partitions:          6
  Partitioner:             metis

Nonlinear System:
  Num DOFs:                1156
  Num Local DOFs:          228
  Variables:               "velocity" { "temperature" "p" }
  Finite Element Types:    "LAGRANGE_VEC" "LAGRANGE"
  Approximation Orders:    "FIRST" "FIRST"

Execution Information:
  Executioner:             Steady
  Solver Mode:             NEWTON
  PETSc Preconditioner:    asm
  MOOSE Preconditioner:    SMP

 0 Nonlinear |R| = 2.883606e+00
      0 Linear |R| = 2.883606e+00
      1 Linear |R| = 1.129123e+00
      2 Linear |R| = 6.765157e-01
      3 Linear |R| = 4.265281e-01
      4 Linear |R| = 2.484776e-01
      5 Linear |R| = 1.716801e-01
      6 Linear |R| = 1.131606e-01
      7 Linear |R| = 5.605766e-02
      8 Linear |R| = 2.447656e-02
      9 Linear |R| = 1.060166e-02
     10 Linear |R| = 5.663119e-03
     11 Linear |R| = 3.970825e-03
     12 Linear |R| = 2.069557e-03
     13 Linear |R| = 1.235481e-03
     14 Linear |R| = 6.456912e-04
     15 Linear |R| = 4.631259e-04
     16 Linear |R| = 4.202744e-04
     17 Linear |R| = 3.971985e-04
     18 Linear |R| = 3.832264e-04
     19 Linear |R| = 3.709511e-04
     20 Linear |R| = 3.631348e-04
     21 Linear |R| = 3.235885e-04
     22 Linear |R| = 1.833163e-04
     23 Linear |R| = 7.047422e-05
     24 Linear |R| = 2.524252e-05
 1 Nonlinear |R| = 1.406771e-02
      0 Linear |R| = 1.406771e-02
      1 Linear |R| = 1.065019e-02
      2 Linear |R| = 6.604399e-03
      3 Linear |R| = 3.995772e-03
      4 Linear |R| = 2.843103e-03
      5 Linear |R| = 2.112097e-03
      6 Linear |R| = 1.571859e-03
      7 Linear |R| = 1.137326e-03
      8 Linear |R| = 6.472439e-04
      9 Linear |R| = 3.574544e-04
     10 Linear |R| = 1.607620e-04
     11 Linear |R| = 6.973255e-05
     12 Linear |R| = 5.012895e-05
     13 Linear |R| = 3.372698e-05
     14 Linear |R| = 2.526075e-05
     15 Linear |R| = 2.335607e-05
     16 Linear |R| = 2.267512e-05
     17 Linear |R| = 2.236576e-05
     18 Linear |R| = 2.191895e-05
     19 Linear |R| = 1.829626e-05
     20 Linear |R| = 1.019462e-05
     21 Linear |R| = 3.101049e-06
     22 Linear |R| = 9.725092e-07
     23 Linear |R| = 4.620495e-07
     24 Linear |R| = 2.434045e-07
     25 Linear |R| = 7.779505e-08
 2 Nonlinear |R| = 2.436942e-05
      0 Linear |R| = 2.436942e-05
      1 Linear |R| = 2.100170e-05
      2 Linear |R| = 1.154197e-05
      3 Linear |R| = 5.889819e-06
      4 Linear |R| = 2.719385e-06
      5 Linear |R| = 1.132952e-06
      6 Linear |R| = 8.574563e-07
      7 Linear |R| = 3.316367e-07
      8 Linear |R| = 1.607685e-07
      9 Linear |R| = 6.271318e-08
     10 Linear |R| = 3.869024e-08
     11 Linear |R| = 2.127475e-08
     12 Linear |R| = 8.533365e-09
     13 Linear |R| = 4.762458e-09
     14 Linear |R| = 2.726666e-09
     15 Linear |R| = 2.456778e-09
     16 Linear |R| = 2.410392e-09
     17 Linear |R| = 2.383263e-09
     18 Linear |R| = 2.324288e-09
     19 Linear |R| = 2.158825e-09
     20 Linear |R| = 1.761493e-09
     21 Linear |R| = 1.276972e-09
     22 Linear |R| = 6.562060e-10
     23 Linear |R| = 3.057528e-10
     24 Linear |R| = 7.837094e-11
 3 Nonlinear |R| = 7.833585e-11
      0 Linear |R| = 7.833585e-11
      1 Linear |R| = 2.727156e-11
      2 Linear |R| = 1.600792e-11
      3 Linear |R| = 1.119906e-11
      4 Linear |R| = 7.506898e-12
      5 Linear |R| = 5.708003e-12
      6 Linear |R| = 4.282397e-12
      7 Linear |R| = 3.432877e-12
      8 Linear |R| = 3.150890e-12
      9 Linear |R| = 3.026838e-12
     10 Linear |R| = 2.970764e-12
     11 Linear |R| = 2.939492e-12
     12 Linear |R| = 2.904253e-12
     13 Linear |R| = 2.792847e-12
     14 Linear |R| = 2.467257e-12
     15 Linear |R| = 1.978193e-12
     16 Linear |R| = 1.375283e-12
     17 Linear |R| = 1.072042e-12
     18 Linear |R| = 7.191070e-13
     19 Linear |R| = 2.080652e-13
     20 Linear |R| = 5.665896e-14
     21 Linear |R| = 1.494566e-14
     22 Linear |R| = 3.960221e-15
     23 Linear |R| = 1.239128e-15
     24 Linear |R| = 2.512554e-16
 4 Nonlinear |R| = 8.845403e-16
 Solve Converged!

Outlier Variable Residual Norms:
  velocity: 8.527382e-16

Now I'm going to try to modify PETSc to support CUDA.