Feature: add abacus toolchain #2857

QuantumMisaka · 2023-08-22T14:04:00Z

Toolchain scripts for quickly and properly compile and install dependencies of ABACUS including:

gnu-toolchain
- gcc (always in system)
- openblas
- openmpi or mpich
- scalapack, fftw, elpa , libxc
- cereal, libnpy, libtorch
intel-toolchain (likely)

Version 2023.1 inspired by cp2k-toolchain.
Still need more test and improvement, but can have good usage.

The toolchain directory is designed to be in the path of ABACUS itself.

Toolchain scripts for quickly and properly compile and install dependencies of ABACUS including: - gnu-toolchain - - `gcc` (always in system) - - `openblas` - - `openmpi` or `mpich` - - `scalapack`, `fftw`, `elpa` , `libxc` - - `cereal`, `libnpy`, `libtorch` - intel-toolchain (likely) Version 2023.1 inspired by cp2k-toolchain. Still need more test and improvement, but can have good usage. The toolchain directory is designed to be in the path of ABACUS itself.

dyzheng · 2023-08-22T14:39:37Z

There are too much new files in this PR. Please check your changes.

too much file mainly in libs/cereal and libs/libnpy, make them easier so less is more

QuantumMisaka · 2023-08-22T22:48:42Z

@dyzheng Got it, much new file mainly in libs/cereal and libs/libnpy, now fix it.

caic99

@QuantumMisaka
Thanks for your contribution! I believe these script will help users a lot.

However, please remove the release packages under libs. Doing so is an action of redistribution and integration of those software packages. This is strongly not recommended because it might introduce incompatibility on licenses. (And you did not include the license files of those packages.)

It is OK to download them in your script. If you think integrating them in ABACUS repo is necessary, please use git submodule.
Thanks!

caic99 · 2023-08-23T02:39:15Z

@QuantumMisaka And, if you wish, you can add docs or links to the toolchain under corresponding parts of abacus-develop/docs in online docs to promote it.

QuantumMisaka · 2023-08-23T02:59:35Z

@caic99 I also want to remove the libraries in libs and use toolchain to install it, I'm trying to do this but It's hard to choose the way to introduce the installation (by git clone or wget from the release, and neither way can bypass github.com, which will bring the HTTP problem). So I highly want to solve this with your help.

Also, the docs file is in plan. I'm wandering if the README file can be directly copy to docs directory,

Thanks !

caic99 · 2023-08-23T03:12:16Z

@caic99 I also want to remove the libraries in libs and use toolchain to install it, I'm trying to do this but It's hard to choose the way to introduce the installation (by git clone or wget from the release, and neither way can bypass github.com, which will bring the HTTP problem). So I highly want to solve this with your help.

In principle it is the user's responsibiliy guaranteeing the internet connection. However, considering the actual situations, maybe you can provide some mirror sites of GitHub (e.g. fastgit) in place of the fixed base domain of github.com . Using gitee mirroring service is a more secure way.

Also, the docs file is in plan. I'm wandering if the README file can be directly copy to docs directory

Please choose a good place for it to maintain the structure. Maybe under quick_start folder, and add an introduction for it at easy_install.md? What do you think?

toolchain/libs/cereal-1.3.2.tar.gz

- use toolchain scripts to install `cereal` and `libnpy` by `wget` from github.com, remove `libs` directory at the same time - fix some bug in intel-toolchain - update README - update executable permission

update readme

…elop into develop

toolchain/README.md

Co-authored-by: Chun Cai <amoycaic@gmail.com>

- update README - update easy_install.md

…elop into develop

add target for acml toolchain

update a quick way to use intel-mpich toolchain installation which has the best speed in test on intel-8358 HPC server

…elop into develop

default gnu toolchain change to openmpi ,which is more efficient in most of the test

- update env setting after installation in `setup` - update build method of intel-toolchain and intel-mpich toolchain - minor bug fixed

use a more normal way to introduce cereal & libnpy

QuantumMisaka · 2023-08-31T15:27:54Z

@caic99 A problem in my try for introducing LibRI in dep to toolchain installation by using intel-icc-toolchain

In file included from /lustre/home/2201110432/apps/abacus/icc-mkl/deps/LibRI/include/RI/global/Tensor.h(8),
                 from /lustre/home/2201110432/apps/abacus/icc-mkl/source/module_hamilt_lcao/hamilt_lcaodft/LCAO_matrix.h(13),
                 from /lustre/home/2201110432/apps/abacus/icc-mkl/source/module_io/restart.h(6),
                 from /lustre/home/2201110432/apps/abacus/icc-mkl/source/module_hamilt_pw/hamilt_pwdft/global.h(8),
                 from /lustre/home/2201110432/apps/abacus/icc-mkl/source/module_cell/klist.cpp(1):
/lustre/home/2201110432/apps/abacus/icc-mkl/deps/LibRI/include/RI/global/Shared_Vector.h(8): catastrophic error: cannot open source file "cereal/cereal.hpp"
  #include <cereal/cereal.hpp>

compilation aborted for /lustre/home/2201110432/apps/abacus/icc-mkl/source/module_cell/klist.cpp (code 4)
gmake[2]: *** [source/module_cell/CMakeFiles/cell.dir/build.make:244: source/module_cell/CMakeFiles/cell.dir/klist.cpp.o] Error 4

What's the best way for LibRI to find CEREAL ?

caic99 · 2023-09-01T02:22:58Z

@caic99 A problem in my try for introducing LibRI in dep to toolchain installation by using intel-icc-toolchain

In file included from /lustre/home/2201110432/apps/abacus/icc-mkl/deps/LibRI/include/RI/global/Tensor.h(8),
                 from /lustre/home/2201110432/apps/abacus/icc-mkl/source/module_hamilt_lcao/hamilt_lcaodft/LCAO_matrix.h(13),
                 from /lustre/home/2201110432/apps/abacus/icc-mkl/source/module_io/restart.h(6),
                 from /lustre/home/2201110432/apps/abacus/icc-mkl/source/module_hamilt_pw/hamilt_pwdft/global.h(8),
                 from /lustre/home/2201110432/apps/abacus/icc-mkl/source/module_cell/klist.cpp(1):
/lustre/home/2201110432/apps/abacus/icc-mkl/deps/LibRI/include/RI/global/Shared_Vector.h(8): catastrophic error: cannot open source file "cereal/cereal.hpp"
  #include <cereal/cereal.hpp>

compilation aborted for /lustre/home/2201110432/apps/abacus/icc-mkl/source/module_cell/klist.cpp (code 4)
gmake[2]: *** [source/module_cell/CMakeFiles/cell.dir/build.make:244: source/module_cell/CMakeFiles/cell.dir/klist.cpp.o] Error 4

What's the best way for LibRI to find CEREAL ?

Hi @QuantumMisaka ,
Please check if LibRI has CEREAL for its git submodule. If not so, you can add CEREAL as an submodule in your toolchain.

caic99 · 2023-09-01T02:26:46Z

Hi @QuantumMisaka , Please check if LibRI has CEREAL for its git submodule. If not so, you can add CEREAL as an submodule in your toolchain.

@QuantumMisaka Update: Please check in your terminal if cereal/cereal.hpp can be opened in editor like vim. If the file is not present, the warning message would be fatal error: xxx.h: No such file or directory, but not catastrophic error: cannot open source file. The latter means the file presents, but the compiler can not open it. Please check file access permission.

…elop into develop

QuantumMisaka · 2023-09-01T03:01:40Z

@QuantumMisaka Update: Please check in your terminal if cereal/cereal.hpp can be opened in editor like vim. If the file is not present, the warning message would be fatal error: xxx.h: No such file or directory, but not catastrophic error: cannot open source file. The latter means the file presents, but the compiler can not open it. Please check file access permission.

The permission of cereal.hpp is 644, which can be open by vim,
The problem still exists after change permission of cereal.hpp to 777
The problem will be fixed if `export CPATH="cereal/include":$CPATH

Libtorch 2.0.1 will lead to lots of warning in abacus build process

make file lesser

6dee7d4

too much file mainly in libs/cereal and libs/libnpy, make them easier so less is more

caic99 requested changes Aug 23, 2023

View reviewed changes

Merge branch 'develop' into develop

c69d449

wenfei-li reviewed Aug 23, 2023

View reviewed changes

toolchain/libs/cereal-1.3.2.tar.gz Outdated Show resolved Hide resolved

QuantumMisaka and others added 4 commits August 23, 2023 22:14

version 2023.2

c17e451

- use toolchain scripts to install `cereal` and `libnpy` by `wget` from github.com, remove `libs` directory at the same time - fix some bug in intel-toolchain - update README - update executable permission

Merge branch 'develop' into develop

8c04886

update README

97c9e76

update readme

Merge branch 'develop' of https://github.com/QuantumMisaka/abacus-dev…

ba4b47e

…elop into develop

caic99 reviewed Aug 23, 2023

View reviewed changes

toolchain/README.md Outdated Show resolved Hide resolved

QuantumMisaka and others added 15 commits August 24, 2023 08:48

Update toolchain/README.md

606fbfc

Co-authored-by: Chun Cai <amoycaic@gmail.com>

Update docs

5d7855e

- update README - update easy_install.md

Merge branch 'develop' of https://github.com/QuantumMisaka/abacus-dev…

0bbc162

…elop into develop

Update README.md

3a6ca74

add target for acml toolchain

Update easy_install.md

dd173d8

update intel-mpich toolchain

5ab58d9

update a quick way to use intel-mpich toolchain installation which has the best speed in test on intel-8358 HPC server

Merge branch 'develop' into develop

6b0b6e8

minor fixed

21b16d2

Merge branch 'develop' of https://github.com/QuantumMisaka/abacus-dev…

9a7e606

…elop into develop

minor fixed and tag a problem

dde576c

Update toolchain_gnu.sh

0372859

default gnu toolchain change to openmpi ,which is more efficient in most of the test

Merge branch 'deepmodeling:develop' into develop

23ae794

version 2023.3

5359ad6

- update env setting after installation in `setup` - update build method of intel-toolchain and intel-mpich toolchain - minor bug fixed

Merge branch 'develop' into develop

8bd2419

Update easy_install.md

f9c49a1

QuantumMisaka added 6 commits August 31, 2023 20:28

minor fixed

a23ab21

Update toolchain output

5a2d522

FIx: Update gcc setup

42941ac

FIx: executable permission

d81ab58

Update: CEREAL & LIBNPY installation

67b6d60

use a more normal way to introduce cereal & libnpy

Change: leave place for LibRI Support

2f47e0c

Merge branch 'develop' into develop

bf90b40

QuantumMisaka mentioned this pull request Sep 1, 2023

Bug for magnetic system calculation compiled from ABACUS by icx-toolchain #2889

Closed

QuantumMisaka added 2 commits September 1, 2023 10:42

Update install_openmpi.sh

5b3dcf9

Merge branch 'develop' of https://github.com/QuantumMisaka/abacus-dev…

7eb7e6e

…elop into develop

QuantumMisaka added 2 commits September 1, 2023 11:10

Fix: Support LibRI installation

e452854

Fix: Downgrade default libtorch to version 1.12.1

abe8480

Libtorch 2.0.1 will lead to lots of warning in abacus build process

caic99 changed the title ~~Toolkits: abacus toolchain~~ Feature: add abacus toolchain Sep 1, 2023

caic99 approved these changes Sep 1, 2023

View reviewed changes

minoe fixed

4af4b47

caic99 merged commit c5ec30e into deepmodeling:develop Sep 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature: add abacus toolchain #2857

Feature: add abacus toolchain #2857

Uh oh!

QuantumMisaka commented Aug 22, 2023

Uh oh!

dyzheng commented Aug 22, 2023

Uh oh!

QuantumMisaka commented Aug 22, 2023

Uh oh!

caic99 left a comment

Uh oh!

caic99 commented Aug 23, 2023

Uh oh!

QuantumMisaka commented Aug 23, 2023

Uh oh!

caic99 commented Aug 23, 2023

Uh oh!

Uh oh!

Uh oh!

QuantumMisaka commented Aug 31, 2023

Uh oh!

caic99 commented Sep 1, 2023

Uh oh!

caic99 commented Sep 1, 2023

Uh oh!

QuantumMisaka commented Sep 1, 2023

Uh oh!

Uh oh!

Feature: add abacus toolchain #2857

Feature: add abacus toolchain #2857

Uh oh!

Conversation

QuantumMisaka commented Aug 22, 2023

Uh oh!

dyzheng commented Aug 22, 2023

Uh oh!

QuantumMisaka commented Aug 22, 2023

Uh oh!

caic99 left a comment

Choose a reason for hiding this comment

Uh oh!

caic99 commented Aug 23, 2023

Uh oh!

QuantumMisaka commented Aug 23, 2023

Uh oh!

caic99 commented Aug 23, 2023

Uh oh!

Uh oh!

Uh oh!

QuantumMisaka commented Aug 31, 2023

Uh oh!

caic99 commented Sep 1, 2023

Uh oh!

caic99 commented Sep 1, 2023

Uh oh!

QuantumMisaka commented Sep 1, 2023

Uh oh!

Uh oh!