conda-forge package?

[ghutchis]

The conda-forge package for xtb has been very useful (e.g., providing Mac binaries) - will there be a package for crest (or bundling crest with the xtb conda-forge channel)?

[pprcht]

Currently that was not planned, but we'll have a look into this. Bundling this with the xtb conda-forge channel may be a good idea indeed.

[awvwgk]

Bundling packages is against the conda-forge philosophy, so not something we should do. Let's plan for a separate crest package depending on the xtb package, conda will care for making the proper install than.

[MDAddario]

Have there been any updates on this? A conda install crest (or through conda-forge) would be extremely helpful for continuous integration and fast setups on remote servers

[awvwgk]

We are not yet there, a few other things have to happen first before we can stage a recipe for conda-forge.

[ghutchis]

@awvwgk @pprcht - anything we can do to help get a crest recipe for condo-forge? (I'd particularly like one for my Mac laptop. 😄 )

[awvwgk]

One step at a time, first we should check if crest can be compiled with GCC or what we have to change to make this possible. Once crest builds with GCC I will submit it to conda-forge.

If you are looking for a version for your Mac, have a look in the Intel oneAPI distribution, which should work on Mac as well to compile crest from source.

[ghutchis]

Now that I have the source, I can try with GCC, but yes, I'll check into the Intel compilers.

[awvwgk]

We are getting closer to a solution here. For a conda-forge package crest will be linked against the netlib BLAS and LAPACK API instead of the MKL to allow exchanging the LAPACK/BLAS backend at runtime. Therefore, crest should build correctly with meson setup _build -Dla_backend=netlib. Currently this leads to linking errors as there are MKL specific routines in use:

/usr/bin/ld: crest.p/src_ompmklset.f90.o: in function `ompautoset_':
/home/awvwgk/projects/src/git/crest/gh-crest/_build/../src/ompmklset.f90:193: undefined reference to `mkl_set_num_threads_'
/usr/bin/ld: crest.p/src_solve.f.o: in function `intens_':
/home/awvwgk/projects/src/git/crest/gh-crest/_build/../src/solve.f:267: undefined reference to `mkl_scoomm_'
/usr/bin/ld: /home/awvwgk/projects/src/git/crest/gh-crest/_build/../src/solve.f:295: undefined reference to `mkl_scoomm_'
collect2: error: ld returned 1 exit status

mkl_set_num_threads_ can be guarded with preprocessor, but scoomm doesn't seem to be part of the BLAS API, but of the sparse BLAS API, which is not available on conda-forge as far as I know. We can always require a specific BLAS/LAPACK implementation in the conda recipe, i.e. enforcing always MKL for crest, but preferably we can avoid this with small changes in the source here.

[awvwgk]

Started a recipe at conda-forge/staged-recipes#15919 for crest.

Linux build is working, but limited to MKL, for OSX we see the following failure:

+ meson --prefix=/Users/runner/miniforge3/conda-bld/crest_1629108083403/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla --libdir=lib --buildtype=release --warnlevel=0 -Dla_backend=mkl -Ddefault_library=shared ..
The Meson build system
Version: 0.59.0
Source dir: $SRC_DIR
Build dir: $SRC_DIR/_build
Build type: native build
Project name: crest
Project version: 2.11.0
Fortran compiler for the host machine: $BUILD_PREFIX/bin/x86_64-apple-darwin13.4.0-gfortran (gcc 9.3.0 "GNU Fortran (GCC) 9.3.0")
Fortran linker for the host machine: $BUILD_PREFIX/bin/x86_64-apple-darwin13.4.0-gfortran ld64 530
C compiler for the host machine: x86_64-apple-darwin13.4.0-clang (clang 11.1.0 "clang version 11.1.0")
C linker for the host machine: x86_64-apple-darwin13.4.0-clang ld64 530
Host machine cpu family: x86_64
Host machine cpu: x86_64
Run-time dependency OpenMP found: YES 5.0

../config/meson.build:70:4: ERROR: Fortran shared or static library 'mkl_gf_lp64' not found

In case somebody has an OSX version around and could try to build crest using conda-forge's MKL and GCC versions, this might be insightful. Otherwise I will just disable OSX support for conda-forge and we can try to figure this out at a later point.

[kjelljorner]

I am able to build with CMake and gfortran-11, gcc-11 and conda-installed MKL on a M1 Mac under Rosetta. Generally I can't get Meson to detect MKL successfully and get the same error as you. Would think it is a Meson problem.

[awvwgk]

Available on conda-forge now, see conda-forge/staged-recipes#16266. Thanks, @jan-janssen.

[jan-janssen]

Sure, if any of you guys want to join me in maintaining the conda package just let me know.

…

On Thu, Oct 7, 2021, 12:49 PM Sebastian Ehlert ***@***.***> wrote: Closed #4 <grimme-lab#4>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <grimme-lab#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5NDE5WQ2REL7RLXIEDOHDUFXTR7ANCNFSM4NGCVIBQ> .

[kjelljorner]

The main issue here for me was the absence of a Mac version on conda-forge. I think that should be possible with the same build.sh that you are using @jan-janssen.

[awvwgk]

Added OSX support in conda-forge/crest-feedstock#3. No support for the new ARM processors though, unless there exist a port of the MKL for OSX/aarch64.

[kjelljorner]

Thanks, much appreciated! As far as I understand, Intel compiler/MKL support for ARM is nonexistent and will remain so for the foreseeable future. Support through Rosetta should be sufficient for now, but long term maybe would be nice with openblas that would enable ARM builds. Personally, I only use it on my Mac for development and would run any major jobs on a Linux cluster anyway.

[awvwgk]

The issue is mainly related to the usage of some MKL specific procedures, like mkl_scoomm from the sparse BLAS API:

https://github.com/grimme-lab/crest/blob/370d6155d0480ef0decaf7945a4398c68452dd2b/src/solve.f#L266-L267

Not sure if there are more MKL specific routines in use. Having a drop-in for those would allow to remove the strict requirement on MKL. This is of course an open source project and contributions are always welcome ;).

[kjelljorner]

I had a look at the subroutine where it is used:
https://github.com/grimme-lab/crest/blob/594ef26f266ec0366e4340a349351a7c7c2de0c0/src/solve.f#L185

It is in turn called by the following subroutine, which as far as I can see is not called at all in the rest of the code.
https://github.com/grimme-lab/crest/blob/594ef26f266ec0366e4340a349351a7c7c2de0c0/src/solve.f#L20

Maybe it is just legacy code that could be removed? If I were to guess, it could be part of the NMR workflow that could have been moved to ANMR?

I did a test by:

1. Commenting out "solve.f" in the src/meson.build
2. Commenting out all calls to MKL_Set_Num_Threadsin https://github.com/grimme-lab/crest/blob/594ef26f266ec0366e4340a349351a7c7c2de0c0/src/ompmklset.f90#L175
3. Building with meson and -Dla_backend=openblas

It compiles without errors and seems to be able to run a CREST job for butane without any problems. I would love to put in a pull request, but not confident how to handle the MKL_Set_Num_Threads calls. Similar to what xtb does?
https://github.com/grimme-lab/xtb/blob/9cc3c208a923f91fd662d132c810976fd089ef3b/src/prog/main.F90#L1186