Skip to content

Commit

Permalink
Dev 0.22 (#1011)
Browse files Browse the repository at this point in the history 
* Set submodule spack to v0.22.2

* fdb: Remove package

* metkit: Remove package

* py-frozendict: Remove package

* xpmem: Remove package

* py-hatchling: Update from upstream

* py-tabulate: Update from upstream

* icon-ham: Use spack/icon

* icon: Split package

* py-lark: Remove package

* Remove unnecessary tests

* Migrate unit tests to pytest

* Rename icon to icon-c2sm

* cosmo: Remove package

* Jenkins: Use mch/prgenv-icon:rc1

* GitHub Action: Apply Pep8-formatting

* Add tests for 'spack find' and 'spack compilers'.

* Start uenv in test stages

* deactivate

* Simplify run_with_spack

* Keep venv

* Bring back devirtualization

* Update uenv

* Use .venv

* GitHub Action: Apply Pep8-formatting

* Shuffle Jenkinsfile

* revert

* uenv

* revert clean env

* Remove fdb-fortran

* flexpart-cosmo: Add workaround

* Update icon-mch tests

* Typo

* icontools: Workaround

* Fix some icon tests

* Fix type

* Remove icon-mch gcc test

* GitHub Action: Apply Pep8-formatting

* Revert "cosmo: Remove package"

This reverts commit 2a7a337.

* cosmo: remove package

* cosmo: remove tests

* Fix icon-mch test

* fix icon test

* eccodes: remove version 2.19.0

* Clean up system tests

* GitHub Action: Apply Pep8-formatting

* Update uenv

* cosmo-eccodes-definitions: Remove versions 2.19.0.x

* icon-mch: Add workarounds

* icon-c2sm: Inherit from icon

* Improve tests

* Disable failing package tests.

* Move mpi workaround

* GitHub Action: Apply Pep8-formatting

* icon-mch: Remove gcc test and add version tests

* icon-c2sm: Remove copy_runscript_related_input_files because there's no self.out_of_source_build

* icon-mch: fix tags

* icon: Set new maintainer

* Rename icon-c2sm to icon

* Use require

* GitHub Action: Apply Pep8-formatting

* pytorch-fortran-proxy: requires gcc

* GitHub Action: Apply Pep8-formatting

* dirty gt4py fix

* GitHub Action: Apply Pep8-formatting

* rm exclaim

* GitHub Action: Apply Pep8-formatting

* fix test and add icon-exlcaim

* GitHub Action: Apply Pep8-formatting

* make out-of-source work

* GitHub Action: Apply Pep8-formatting

* fix url

* rm file

* update test

* cleanup

* GitHub Action: Apply Pep8-formatting

* fix

* fix version

* rm include

* GitHub Action: Apply Pep8-formatting

* fix icon-ham

* add maintainer

* Update repos/c2sm/packages/icon-exclaim/package.py

Co-authored-by: Dominic Hofer <6570912+dominichofer@users.noreply.github.com>

* GitHub Action: Apply Pep8-formatting

* change

* GitHub Action: Apply Pep8-formatting

* disable version

* GitHub Action: Apply Pep8-formatting

* remove loop-exchange

* fix icon

* dirty fix icon4py

* readd loop-exchange

* Update package.py

Set default loop-exchange to False

* fix bug again of serialbox dep, cleanup

* reintroduce skip configure

* remove libtorch

* upgrade docs (#1038)

* upgrade docs

* add yaml example

* update docs

* update docs

* Update cosmo-eccodes-definitions versions to 2.36.0.3
This is used with eccodes 2.36.4
This is required to test v8 env on balfrin

* use explicit zlib-ng dependency for icon (#1040)

* use rc 4

* remove obsolete packages

* adjust integration tests

* upgrade manual to build icon

---------

Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: juckerj <39263956+jonasjucker@users.noreply.github.com>
Co-authored-by: Daniel Hupp <daniel.hupp@meteoswiss.ch>
Co-authored-by: Jonas Jucker <jonas.jucker@c2sm.ethz.ch>
Co-authored-by: Xavier Lapillonne <7485594+lxavier@users.noreply.github.com>
Co-authored-by: Ben Cumming <bcumming@cscs.ch>
  • Loading branch information
@dominichofer @jonasjucker
@huppd @lxavier @bcumming
7 people authored Jan 28, 2025
1 parent e5479e6 commit 6ed9b0f
Show file tree
Hide file tree
Showing 42 changed files with 379 additions and 1,988 deletions.
34 changes: 23 additions & 11 deletions Jenkinsfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,20 +20,30 @@ pipeline {
}
}
stages {
stage('Create environment') {
stage('Create python environment') {
steps {
sh """
python3 -m venv env
source env/bin/activate
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
"""
}
}
stage('Create uenv') {
steps {
sh """
git clone -b fix/jenkins https://github.com/eth-cscs/uenv.git
./uenv/install --yes --destdir=$WORKSPACE
source $WORKSPACE/etc/profile.d/uenv.sh
uenv repo create
uenv image pull mch/v8:rc4
"""
}
}
stage('Bootstrap spack') {
// Bootstrapping spack is a separate stage to avoid problems with concurrently bootstrapping spack in the tests.
steps {
sh """
source env/bin/activate
source ./setup-env.sh
spack spec gnuconfig
"""
Expand All @@ -42,26 +52,28 @@ pipeline {
stage('Unit Tests') {
steps {
sh """
source env/bin/activate
source .venv/bin/activate
python3 test/unit_test.py
"""
}
}
stage('Integration Tests') {
steps {
sh """
source env/bin/activate
source ./setup-env.sh /mch-environment/v7
pytest -v -n auto test/integration_test.py
source $WORKSPACE/etc/profile.d/uenv.sh
source ./setup-env.sh /user-environment
source .venv/bin/activate
uenv run mch/v8:rc4 -- pytest -v -n auto test/integration_test.py
"""
}
}
stage('System Tests') {
steps {
sh """
source env/bin/activate
source ./setup-env.sh /mch-environment/v7
pytest -v -n auto test/system_test.py
source $WORKSPACE/etc/profile.d/uenv.sh
source ./setup-env.sh /user-environment
source .venv/bin/activate
uenv run mch/v8:rc4 -- pytest -v -n auto test/system_test.py
"""
}
}
Expand Down
48 changes: 12 additions & 36 deletions docs/CodeDevelopment.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,42 +3,6 @@ Develop packages

Spack offers several options for package development.
Depending on your workflow, one or the other option is preferred.
Also some packages like ICON or COSMO have their own
development workflow which is maintained by C2SM.

Plain dev-build
---------------

This is the easiest way to build local sources.
Enter the root of your source repository and execute:

.. code-block:: console
$ spack dev-build --until build <package> @<version>
This will build the package as is. The downside of this approach is that
you need to go through all phases of a package build.

Dev-build in combination with build-env
---------------------------------------

We assume that developers of a package are familiar with its build system.
Therefore, we reccomend to use Spack to set up the environment for the package.
Building and testing should be done with the package's build and test system.

.. code-block:: console
# Load Spack!
$ spack dev-build --before build <package> @develop <variant> # stops dev-build before executing the phase 'build'
$ spack build-env <package> @develop <variant> -- bash # nests a bash shell with the build env vars loaded
# Work on the package!
# Use the build system of the package! (e.g. 'make')
# Use the testing infrastructure of the package!
$ exit # to exit the nested bash
If you want multiple dev-builds at the same time, label them with separate ``@<your-label>``.
The identifier ``@develop`` is common in the Spack documentation but you can use any string.


Environments with Spack develop
-------------------------------
Expand Down Expand Up @@ -95,3 +59,15 @@ To deactivate a Spack environment, type
Most of the Spack commands are sensitive to environments, see
`Spack docs <https://spack.readthedocs.io/en/latest/environments.html#environment-sensitive-commands>`__.

Plain dev-build
---------------

This is the easiest way to build local sources.
Enter the root of your source repository and execute:

.. code-block:: console
$ spack dev-build --until build <package> @<version>
This will build the package as is. The downside of this approach is that
you need to go through all phases of a package build.
71 changes: 34 additions & 37 deletions docs/QuickStart.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,19 +11,17 @@ To set up a Spack instance, clone the repository using a specific Spack tag (lat
$ git clone --depth 1 --recurse-submodules --shallow-submodules -b $SPACK_TAG https://github.com/C2SM/spack-c2sm.git
To load it into your command line, execute
To load it into your command line, execute one of the following commands:

.. code-block:: console
$ . spack-c2sm/setup-env.sh
$ . spack-c2sm/setup-env.sh /user-environment
$ . spack-c2sm/setup-env.sh /mch-environment/v6
$ . spack-c2sm/setup-env.sh /mch-environment/v7
$ . spack-c2sm/setup-env.sh any_other_upstream
This auto-detects your machine and configures your instance for it.
You can force a machine with an argument. The name has to match a folder in sysconfigs.

.. code-block:: console
$ . spack-c2sm/setup-env.sh balfrin
This will make upstream installation from user-environment available in spack-c2sm.

Local machines and Containers
-----------------------------
Expand Down Expand Up @@ -84,21 +82,19 @@ ICON
----

ICON is built using environments.

For convenience, ICON provides bash scripts to set up the environment and install ICON for in-source
and out-of-source builds.
These scripts are located in ``config/cscs``, e.g. ``config/cscs/alps_mch.cpu.nvidia``.

For development, sometimes it is necessary to build ICON in a more customized way.
To do so please follow the instructions below.

Environments are located in a folder named after the environment and are defined in a ``spack.yaml`` file.
For ICON, they are located in ``config/cscs/spack/<version>/<machine>_<target>_<compiler>``.
For ICON, they are located in ``config/cscs/spack/<machine>_<target>_<compiler>``.
They work with a special Spack tag, that is provided in the ICON repository at ``config/cscs/SPACK_TAG_*``.
So make sure you clone Spack with the specified tag.

.. tip::
**On Balfrin:**
In case your Spack environment requires Python, a compatability issue
with `openssl` and `git` appears.

``/usr/bin/ssh: symbol lookup error: /usr/bin/ssh: undefined symbol: EVP_KDF_CTX_free, version OPENSSL_1_1_1d``

To circumvent that simply do
``spack load git`` prior to activation of the environment.

To activate the Spack environment, type

.. code-block:: console
Expand All @@ -109,14 +105,16 @@ To install the environment and so ICON, type

.. code-block:: console
$ spack develop --path $(pwd) icon@develop
$ spack install
Example to build ICON for CPU with NVHPC on Piz Daint:
Example to build ICON for CPU with NVHPC on Balfrin:

.. code-block:: console
$ SPACK_TAG=$(cat "config/cscs/SPACK_TAG_DAINT")
$ spack env activate -d config/cscs/spack/$SPACK_TAG/daint_cpu_nvhpc
$ SPACK_TAG=$(cat "config/cscs/SPACK_TAG_MCH")
$ spack env activate -d config/cscs/spack/mch_cpu_double
$ spack develop --path $(pwd) icon@develop
$ spack install
.. attention::
Expand All @@ -127,23 +125,22 @@ Out-of-source builds are possible as follows:

.. code-block:: console
$ mkdir cpu && cd cpu
$ cp -r ../config .
$ spack env activate -d config/cscs/spack/v0.20.1.5/daint_cpu_nvhpc
$ mkdir cpu
$ spack env activate -d config/cscs/spack/mch_cpu_double
$ # tell spack to build icon in folder cpu
$ spack develop --path $(pwd) --build-directory cpu icon@develop
$ spack install
.. attention::
Out-of-source build for AutotoolsPackages is not supported by Spack.
The implementation for ICON relies on some hacks inside package.py and
only works if the build-folder is located inside the Git repo of ICON.

COSMO
-----

Building COSMO is not supported anymore starting with spack-c2sm v0.20.1.0 for CSCS.
By executing the commands above, spack will add some lines directly into ``spack.yaml``:

For Euler Cluster a CPU-only setup is supported using GCC 8.5.0
.. code-block:: yaml
.. code-block:: console
spack:
packages:
icon:
package_attributes:
build_directory: /scratch/mch/juckerj/icon-nwp/cpu
$ spack install cosmo%gcc@8.5.0
Any further ``spack install`` command will use the build directory specified in the ``spack.yaml`` file.
In case you want to change the build directory, edit the ``spack.yaml`` file or remove the ``build_directory`` line
and run ``spack concretize -f`` afterwards.
40 changes: 6 additions & 34 deletions repos/c2sm/packages/cosmo-eccodes-definitions/package.py
View file
Original file line number Diff line number Diff line change
@@ -1,25 +1,3 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)

# ----------------------------------------------------------------------------
# If you submit this package back to Spack as a pull request,
# please first remove this boilerplate and all FIXME comments.
#
# This is a template package file for Spack. We've put "FIXME"
# next to all the things you'll want to change. Once you've handled
# them, you can save this file and test your package like this:
#
# spack install cosmo-eccodes-definitions
#
# You can edit this file again by typing:
#
# spack edit cosmo-eccodes-definitions
#
# See the Spack documentation for more information on packaging.
# ----------------------------------------------------------------------------

from spack import *


Expand All @@ -30,27 +8,21 @@ class CosmoEccodesDefinitions(Package):
url = "https://github.com/COSMO-ORG/eccodes-cosmo-resources.git"
git = 'https://github.com/COSMO-ORG/eccodes-cosmo-resources.git'

maintainers = ['petrabaumann']
maintainers = ['huppd,lxavier']

version('2.36.0.3', tag='v2.36.0.3')
version('2.25.0.3', tag='v2.25.0.3')
version('2.25.0.2', tag='v2.25.0.2')
version('2.25.0.1', tag='v2.25.0.1')
version('2.19.0.7', tag='v2.19.0.7')
version('2.19.0.6', tag='v2.19.0.6')
version('2.19.0.5', tag='v2.19.0.5')
version('2.19.0.4', tag='v2.19.0.4')
version('2.19.0.3', tag='v2.19.0.3')
version('2.19.0.2', tag='v2.19.0.2')
version('2.19.0.1', tag='v2.19.0.1')
version('2.18.0.1', tag='v2.18.0.1')

depends_on('eccodes')
depends_on('eccodes@2.25.0',
depends_on('eccodes@2.36.4',
type=('build', 'link', 'run'),
when='@2.25.0.1:')
depends_on('eccodes@2.19.0',
when='@2.36.0.3')
depends_on('eccodes@2.25.0',
type=('build', 'link', 'run'),
when='@2.19.0.1:2.19.0.7')
when='@2.25.0.1:2.25.0.3')
depends_on('eccodes@2.18.0',
type=('build', 'link', 'run'),
when='@2.18.0.1')
Expand Down
Loading

0 comments on commit 6ed9b0f

Please sign in to comment.