Packages on Narwhal on Nautilus

[areinecke]

Using this form because the Install/Upgrade form only allows to request a single package.

On nautilus and narwhal we need to the following available for NEPTUNE:

MCT
ESMF 8.5
FFTW 3.3.9
LAPACK 3.11.0

[climbfuji]

I am looking at spack-stack-1.6.0 on Nautilus. It has:

mct@2.11.0
esmf@8.5.0
esmf@8.6.0

However, we don't install fftw and lapack on Nautilus, because in a previous conversation we agreed that we'd be using MKL as the virtual provider for fftw, blas and lapack. Thus, the entire software stack is built on using MKL for consistency.

Do you want to use the "real" fftw and lapack packages instead of mkl throughout, or do you want to have the stack use mkl as it is now but still have the other packages available?

Note that the story is different on Narwhal, I think we still use openblas as the provider for blas and lapack, and fftw (i.e. no mkl).

[climbfuji]

On Nautilus:

mct@2.11.0
esmf@8.5.0
esmf@8.6.0
fftw@3.3.10
openblas@0.3.24 - provides blas and lapack

[climbfuji]

So the main question for me is "does it have to be the original lapack and fftw" or do (mkl OR (fftw + openblas)) suffice?

[climbfuji]

Last one for now, I promise. I checked and we can switch to using the original netlib lapack as provider for blas and lapack, if openblas or mkl are not sufficient.

[unified-env-test] ubuntu@ip-10-0-1-144:/mnt/experiments-zfs/ubuntu/spack-stack-fix-ci-rnd$ spack providers blas
blas:
acfl  amdblis  amdblis  armpl-gcc  atlas  blis  blis  clblast  cray-libsci  essl  flexiblas  fujitsu-ssl2  intel-mkl  intel-oneapi-mkl  intel-parallel-studio  netlib-lapack  netlib-xblas  nvhpc  openblas  veclibfort

[unified-env-test] ubuntu@ip-10-0-1-144:/mnt/experiments-zfs/ubuntu/spack-stack-fix-ci-rnd$ spack info netlib-lapack
CMakePackage:   netlib-lapack

Description:
    LAPACK version 3.X is a comprehensive FORTRAN library that does linear
    algebra operations including matrix inversions, least squared solutions
    to linear sets of equations, eigenvector analysis, singular value
    decomposition, etc. It is a very comprehensive and reputable package
    that has found extensive use in the scientific community.

Homepage: https://www.netlib.org/lapack/

Preferred version:
    3.11.0    https://github.com/Reference-LAPACK/lapack/archive/refs/tags/v3.11.0.tar.gz

Safe versions:
    3.11.0    https://github.com/Reference-LAPACK/lapack/archive/refs/tags/v3.11.0.tar.gz
    3.10.1    https://github.com/Reference-LAPACK/lapack/archive/refs/tags/v3.10.1.tar.gz
    3.10.0    https://github.com/Reference-LAPACK/lapack/archive/refs/tags/v3.10.0.tar.gz
    3.9.1     https://github.com/Reference-LAPACK/lapack/archive/refs/tags/v3.9.1.tar.gz
    3.9.0     https://github.com/Reference-LAPACK/lapack/archive/v3.9.0.tar.gz
    3.8.0     https://www.netlib.org/lapack/lapack-3.8.0.tar.gz
    3.7.1     https://www.netlib.org/lapack/lapack-3.7.1.tgz
    3.7.0     https://www.netlib.org/lapack/lapack-3.7.0.tgz
    3.6.1     https://www.netlib.org/lapack/lapack-3.6.1.tgz
    3.6.0     https://www.netlib.org/lapack/lapack-3.6.0.tgz
    3.5.0     https://www.netlib.org/lapack/lapack-3.5.0.tgz
    3.4.2     https://www.netlib.org/lapack/lapack-3.4.2.tgz
    3.4.1     https://www.netlib.org/lapack/lapack-3.4.1.tgz
    3.4.0     https://www.netlib.org/lapack/lapack-3.4.0.tgz
    3.3.1     https://www.netlib.org/lapack/lapack-3.3.1.tgz

Deprecated versions:
    None

Variants:
    build_system [cmake]         cmake
        Build systems supported by the package
    external-blas [false]        false, true
        Build lapack with an external blas
    lapacke [true]               false, true
        Activates the build of the LAPACKE C interface
    shared [true]                false, true
        Build shared library version
    xblas [false]                false, true
        Builds extended precision routines using XBLAS

    when build_system=cmake
      build_type [Release]       Debug, MinSizeRel, RelWithDebInfo, Release
          CMake build type
      generator [make]           none
          the build system generator to use

    when build_system=cmake ^cmake@3.9:
      ipo [false]                false, true
          CMake interprocedural optimization

Build Dependencies:
    blas  cmake  gmake  netlib-xblas  ninja

Link Dependencies:
    blas  netlib-xblas

Run Dependencies:
    None

Licenses:
    None

[areinecke]

Sorry, for the delay, for some reason I don't git email notifications from github.

Regarding fftw and lapack, it's a requirement of the ESPC system. I think they just need it available but not part of the stack build.

I saw that 1.6 was there but can you modify the permissions.

[climbfuji]

Sorry, for the delay, for some reason I don't git email notifications from github.

Regarding fftw and lapack, it's a requirement of the ESPC system. I think they just need it available but not part of the stack build.

I saw that 1.6 was there but can you modify the permissions.

Sorry for the wrong permissions ... we do set umask before we install spack-stack, but for some reason that doesn't get honored (or gets reverted?). Anyway, I fixed it on Narwhal and Nautilus just now.

Let me try to add fftw and lapack in a test environment first to see if it confuses the spack concretizer or not, and if yes how to get around it.

[climbfuji]

@areinecke I installed fftw@3.3.10 and netlib-lapack@3.11.0 in the existing spack-stack@1.6.0 unified environment on Nautilus for you to check. They don't get loaded automatically when you load jedi-neptune-env, since they are not a build requirement for JEDI-NEPTUNE. One needs to load them explicitly when needed.

I also created PRs for spack and spack-stack for the latest develop code: JCSDA/spack#391 and #947 - this way it's going to be included in the same way in future releases.

Please let me know if this works for you and I will repeat the install in spack-stack@1.6.0 on Narwhal.

[areinecke]

Thanks, @climbfuji. I'll pass the word to the ESPC developers to test it out.

One more issue regarding spack-stack-1.6.0 on nautilus. The sp/2.5.0 library is currently installed under lib64 but was previously under lib. Other ncep libs (bacio, w3nco) are in lib. Could a link from lib to lib64 be added?

[climbfuji]

@AlexanderRichert-NOAA Please note Alex's comment above on the change of the lib subdirectory name for sp@2.5.0 (now it is lib64). Was that intentional? If yes, (when) are the other NCEPLIBS updated to use lib64? Should we, for the time being, create the symlink so that users have more time to adjust their scripts?

[AlexanderRichert-NOAA]

Yeah, so basically we use include(GNUInstallDirs), which dynamically determines whether to call it lib or lib64, whereas previously we were hardcoding 'lib' into the CMake config. All current versions of NCEPLIBS (at least in their develop branches, I haven't checked all the individual versions) use this same approach, i.e., it's determined by CMake according to the GNUInstallDirs schema. Please note that w3nco is no long maintained, so I strongly encourage switching to w3emc as soon as possible.

All that said, I have no objection to putting in symlinks for at least one release to help users transition.

[areinecke]

A symbolic link would be helpful.

We only depend on w3nco because it's required to build the ccpp.

[climbfuji]

A symbolic link would be helpful.

We only depend on w3nco because it's required to build the ccpp.

I think CCPP (latest) has moved on but I need to confirm. In the meantime, we can add the symlink on Nautilus and Narwhal, but if all NCEPLIBS are moving to GNUInstallDirs in the future it would be good to adjust the workflows rather than providing backward compatibility in the future?

[climbfuji]

Reopening - will close when I added the links for sp on Nautilus and Narwhal

[climbfuji]

I created the symlinks in the sp installs on Nautilus (Intel) and Narwhal (Intel and GNU). I also installed fftw and lapack on Narwhal for Intel and GNU, in addition to what I did on Nautilus (Intel) earlier this week.