Skip to content

Conference call notes 20240605

Jump to bottom
Alex Domingo edited this page Jun 5, 2024 · 3 revisions

(back to Conference calls)

Notes on the 247th EasyBuild conference call, Wednesday 05 Jun 2024 (15:00 UTC)

Attendees

List of attendees (10):

  • Sebastian Achilles (JSC, Germany)
  • Alex Domingo (Vrije Universiteit Brussel, Belgium)
  • Jasper Grimm (University of York, UK)
  • Em Dragowsky
  • Bart Oldeman
  • Jörg Saßmannshausen (Imperial College London, UK)
  • Simon Branford
  • Kurt Lust (UAntwerpen, Belgium + LUMI User Support Team)
  • Mikael Öhman (Chalmers University of Technology, Sweden)
  • Sam Moors (Vrije Universiteit Brussel, Belgium)

Agenda

  • overview of recent developments
  • outlook to EasyBuild 5.0 release
  • 2024a update of common toolchains
  • Q&A

Recent developments

  • latest EasyBuild release: 4.9.1 (3 Apr 2024)
  • next EasyBuild release: probably 4.9.2 in coming weeks, to help with transition to 5.0.0
  • EasyBuild v5.0.0
    • detailed notes on latest developments in https://hackmd.io/tZtz6vMmRfGL6N9Rf__OEA
    • to test development version of EasyBuild 5.0: 
      # set up Python virtual environment, and jump into it
python3 -m venv eb5
source eb5/bin/activate

# install EasyBuild 5.0 development version into it
pip install https://github.com/easybuilders/easybuild-framework/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyblocks/archive/5.0.x.tar.gz
pip install https://github.com/easybuilders/easybuild-easyconfigs/archive/5.0.x.tar.gz

# go!
eb --version

Merged PRs

  • docs (merged PRs)

    • ...

  • framework (merged PRs)

    • bug fixes
      • Fix resolved (template) values in case of failure (PR #4532)
    • enhancements
      • versions checks to avoid mixing major versions across the EasyBuild components (PR #4520)
        • will help to make clear why --from-pr is broken when using EasyBuild 4.x when 5.0.x branch is collapsed into develop branch (which will happen soon, after release of EasyBuild v4.9.2)
      • add support for easyconfig parameter module_only (PR #4537)
    • changes
      • ...
    • code cleanup
      • ...
    • EasyBuild 5.0 (5.0.x branch)
      • Use dict.items() instead of repeatedly getting the value (PR #4533)
      • create env.sh and cmd.sh helper scripts in run_shell_cmd (framework PR #4486)
      • add support for alternate easyconfig parameters/templates/constants (PR #4511)
        • actual renaming is being done in PR #4514 (which is about to get merged)
        • policy will be documented w.r.t. naming of things:
            1. Multiple words are separated by _ (or - for command line configuration options), as opposed to just concatenating them
            • examples: build_opts (not buildopts), post_install_cmds (not postinstallcmds)
            • exceptions: symlink, sysroot
            1. Common long words can be abbreviated, but only if there's no room for confusion
            • examples: deps (instead of dependencies), opts (instead of options), env_mod (instead of environment_module)
            • exceptions:
              • configure in easyconfig parameters like configure_opts (to be consistent with configure step)
              • cfg will be used EasyBuild configuration options
            1. Confusion with commonly used terms (like "module") is avoided by being more descriptive
            • example: don't use mod for module, but py_mod for Python module, env_mod for environment module, fortran_mod for Fortran module
        • add new get_cwd function to tools.filetools to retrieve current working directory (PR #4525)
        • --fix-deprecated-easyconfigs will be enhanced to auto-rename stuff in easyconfigs (see WIP PR #4547)
      • fix test_update_branch_github which got broken because toy-0.0.eb easyconfig was modified (framework PR #4545)

  • easyblocks (merged PRs)

    • bug fixes
      • ...
    • enhancements
      • allow version mismatchs between OpenSSL components in host system as long as they fulfill version requirements (PR #3340)
      • Allow external PRRTE in OpenMPI easyblock (PR #3347)
    • updates
      • ...
    • changes
      • ...
    • new easyblocks
      • ...
    • EasyBuild 5.0
      • set CMake installation LIBDIR to lib by default in CMakeMake easyblock (PR #3227)

  • easyconfigs (merged PRs)

    • ~100 easyconfig PRs were merged since last conf call
    • bug fixes/reports
      • use PSM2 dependency in libfabric only on x86_64 PR#20585
      • bump async-timeout to 4.0.3 in aiohttp 3.8.5 (PR#20553)
    • enhancements
      • ...
    • (noteworthy) new software
      • ...
    • noteworthy software updates

      • Laying ground in GCCcore/13.3.0 (2024a)

        • GCC v13.3.0 PR#20559
        • Autotools v20231222, Autoconf v2.72, Automake v1.16.5, libtool v2.4.7, Perl v5.38.2 PR#20570
        • pkgconf v2.2.0 PR#20572
        • gettext v0.22.5, XZ v5.4.5, libxml2 v2.12.7, libiconv v1.17, ncurses v6.5 PR#20573
        • CMake v3.29.3, bzip2 v1.0.8, cURL v8.7.1, libarchive v3.7.4, UnZip v6.0 PR#20574
        • numactl v2.0.18, libfabric v1.21.0, PSM2 v12.0.1 PR#20575
        • libevent v2.1.12 PR#20576
        • Python v3.12.3, libreadline v8.2, Tcl v8.6.14, libffi v3.4.5, SQLite v3.45.3 PR#20579
        • Ninja v1.12.1, Meson v1.4.0, libpciaccess v0.18.1 PR#20580
        • hwloc v2.10.0, UCX v1.16.0, UCC v1.3.0 PR#20581
        • xorg-macros v1.20.1 PR#20583
        • make v4.4.1, BLIS v1.0 PR#20608
        • OpenBLAS v0.3.27 PR#20621
        • expat v2.6.2, git v2.45.1 PR#20628
        • cffi v1.16.0, gzip v1.13, libyaml v0.2.5 PR#20629
        • Rust v1.78.0, patchelf v0.18.0 PR#20631
        • gperf v3.1, libpng v1.6.43, util-linux v2.40 PR#20632
        • Doxygen v1.11.0, Eigen v3.4.0, GDRCopy v2.4.1 PR#20633
        • jbigkit v2.1, LibTIFF v4.6.0, libdeflate v1.20 PR#20635
        • mold v2.31.0 PR#20637
        • ICU v75.1, nodejs v20.13.1 PR#20638
        • FlexiBLAS v3.4.4 PR#20643
        • GDB v14.2, ISL v0.26, MPC v1.3.1 PR#20644
        • FFTW v3.3.10, gfbf v2024.05 PR#20645
        • foss v2024.05, gompi v2024.05, FFTW.MPI v3.3.10 PR#20646

      • Jupyter ecosystem in GCCcore/13.2.0 (2023b)

        • tornado v6.4 PR#20563
        • JupyterLab v4.2.0, JupyterNotebook v7.2.0, jupyter-server v2.14.0, hatch-jupyter-builder v0.9.1 PR#20606
        • JupyterHub v4.1.5, bcrypt v4.1.3, configurable-http-proxy v4.6.1 PR#20612
        • Jupyter-bundle v20240522, nbclassic v1.0.0, jupyter-server-proxy v4.1.2, aiohttp v3.9.5, jupyter-resource-usage v1.0.2, jupyter-server-proxy v4.1.2, nbclassic v1.0.0 PR#20613
        • ipympl v0.9.4 PR#20626
        • dask v2024.5.1, bokeh v3.4.1 PR#20657
        • dask-labextension v7.0.0 PR#20658

      • OCaml v5.1.1 (PR#20552)

Open (active) PRs

  • docs (open PRs + issues)

    • ...

  • framework (open PRs + issues)

  • easyblocks (open PRs + issues)

    • bug fixes
      • fix failing sanity check for intel-compiler on Debian/Ubuntu (PR #3346)
      • avoid sanity check on the software itself for CargoPythonBundle easyblock (PR #3341)
    • enhancements
      • enhance CMakeMake easyblock to check whether correct Python installation was picked up by cmake (PR #3233)
      • add Arm support to NAMD easyblock (PR #3348)
    • changes
      • Merge of the ConfigureMake and CMakeMake versions of the easyblock for QuantumESPRESSO (PR #3338)
    • new
      • ...

  • easyconfigs (open PRs + issues)

    • bug fixes
      • ...
    • enhancements
      • ...
    • (noteworthy) new software
      • QuantumESPRESSO:
        • 7.3.1 with cmake (PR #20138)
        • Correct configuration of QuantumESPRESSO for 6.7/7.2/7.3, move all QE easyconfigs to "legacy" easyblock (PR #20070)
      • jax 0.4.25 w/ CUDA (PR #20119)
      • RELION v5.0.0_beta.2024.0.01 w/ CUDA 12.1.1 9 ([PR #19678](https://gi
      • thub.com/easybuilders/easybuild-easyconfigs/pull/19678))
      • PyTorch v2.3.0 (PR #20489)
        • Alexander is in touch with PyTorch devs on getting machine-readable output for PyTorch test suite
    • changes
      • ...

2024a common toolchains

  • GCC 13.3 as a base
  • latest version of all components (except GCC, not using 14.1):
    • foss/2024a (candidate toolchain foss/2024.05 merged: PR #20646)
      • GCC 13.3 (20240521) + binutils 2.42 (20240129) (PR #20559)
      • OpenMPI 5.0.3 (20240408) (PR #17561)
        • PMIx 5.0.2 (20240321), UCX 1.16.0 (20240415), libfabric 1.21.0 (20240329)
      • FlexiBLAS 3.4.3 (20240521) + OpenBLAS 0.3.27 (20240404) + BLIS 1.0 (20240506)
      • FFTW 3.3.10 (20210914)
      • ScaLAPACK 2.2.0 (20220202)
    • intel/2024a (TODO: set up candidate toolchain intel/2024.04?)
      • GCCcore 13.3 + binutils 2.42
      • Intel C/C++/Fortran compilers 2024.1.0 (20240327)
      • Intel MPI 2021.12.1 (20240424)
      • Intel MKL 2024.1.0 (20240327)
  • notes
    • Python 3.11.x or 3.12.x => trying with Python 3.12 for now
    • Sam: Cython should be a build dependency for Python-bundle-PyPI

Q&A / others

  • Jorg: python-javabridge in CellProfiler is not working PR#20725

    • cloning with git_config will not help, as it will result in the same tarball already configure to be pulled from github
    • issue is a typo in source_urls extention parameter

  • Jorg: what is the recommnded solution to build optimized code with Intel compilers on non-Intel platforms

    • optarch for Intel compilers on AMD hardware has to be done manually
    • optimal optarch for zen4 is rocketlake
    • Jasper: we could have a list of optarch per archs hardcoded somewhere in EB

  • Bart: new version Intel C/C++ compiler v2024.1.2, but not for Fortran. However installation of C/C++ compiler with Fortran compiler that have different versions fails.

    • we can wait for the same update on Fortran side
    • issue lies on Intel compilers side, so difficult to fix by us
    • Jasper: if you download the bundle, you get v2024.1.0 for all compilers (C++ and Fortran), we should stick to this combination as well

  • Bart: opinions on PR#4513 get_software_libdir: return dir if it's only 1 with libs?

    • looks good, approach is reasonable but we have to check uses of get_software_libdir in other easyblocks

  • Jasper: what linker will be used in 2024a?

    • mold has its easyconfig merged in GCCcore/13.3.0, but foss/2024.05 continues to be based on good old binutils and nothing else
    • Simon: developer of mold suggested some time ago to change licensing to a non fully open-source model
Clone this wiki locally