From d3c9ac697c9413eccfb8c7939189de1dda93cde3 Mon Sep 17 00:00:00 2001 From: Sebastiaan Huber Date: Wed, 12 Jan 2022 13:10:20 +0100 Subject: [PATCH] Dependencies: explicitly compile `pymatgen` with compatible `numpy` With the release of `numpy==1.22.0` on December 31 2022, our CI builds started failing. The reason being that `pymatgen` would get built with a version of `numpy` that is incompatible with the version of `numpy` that would be installed in the environment in which the tests are run. The reason for this is that `pymatgen` does not specify an explicit version of `numpy` in its build requirements and so will pick the latest version. But `numpy` only guarantees foreward compatibility with its ABI and so if compiled with a certain version, the resulting binary won't be compatible with older `numpy` versions. In this case explicitly, `pymatgen` was getting built with `numpy==1.22` but then the tests were run with `1.21.4` which would cause the exception. The solution is to ensure `pymatgen` is compiled with a version of `numpy` that is ABI-compatible with the version that will be used in the actual test run. The way to do this is to first explicitly install a version of `numpy` that we need and then install `pymatgen` using the `--no-build-isolation` flag to force `pip` to not install build requirements but take them from the existing environment. This is also why we need to install the `wheel` package explicitly, because this is also a build requirement of `pymatgen` that no longer will be installed automatically by `pip` due to `--no-build-isolation`. Finally, there was a necessary change for the dependencies of Python 3.10 where we update the version of `pymatgen` because older versions could not be built since the building used CPython code that has been removed in Python 3.10. --- .github/workflows/benchmark.yml | 23 ++++++++++++++++++++++- .github/workflows/ci-code.yml | 19 +++++++++++++++++-- .github/workflows/rabbitmq.yml | 16 ++++++++++++++++ .github/workflows/release.yml | 23 ++++++++++++++++++++++- requirements/requirements-py-3.10.txt | 4 ++-- setup.json | 2 +- 6 files changed, 80 insertions(+), 7 deletions(-) diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml index 1d9368b82d..7afb6c7b30 100644 --- a/.github/workflows/benchmark.yml +++ b/.github/workflows/benchmark.yml @@ -50,9 +50,30 @@ jobs: uses: actions/setup-python@v2 with: python-version: '3.8' + + - name: Upgrade pip + run: | + pip install --upgrade pip + pip --version + + - name: Build pymatgen with compatible numpy + run: | + # This step is necessary because certain versions of `pymatgen` will not specify an explicit version of + # `numpy` in its build requirements, and so the latest version will be used. This causes problems, + # however, because this means that the compiled version of `pymatgen` can only be used with that version + # of `numpy` or higher, since `numpy` only guarantees forward compatibility of the ABI. If we want to + # run with an older version of `numpy`, we need to ensure that `pymatgen` is built with that same + # version. This we can accomplish by installing the desired version of `numpy` manually and then calling + # the install command for `pymatgen` with the `--no-build-isolation` flag. This flag will ensure that + # build dependencies are ignored and won't be installed (preventing the most recent version of `numpy` + # to be installed) and the build relies on those requirements already being present in the environment. + # We also need to install `wheel` because otherwise the `pymatgen` build will fail because `bdist_wheel` + # will not be available. + pip install numpy==1.21.4 wheel + pip install pymatgen==2022.0.16 --no-cache-dir --no-build-isolation + - name: Install python dependencies run: | - python -m pip install --upgrade pip pip install -r requirements/requirements-py-3.8.txt pip install --no-deps -e . pip freeze diff --git a/.github/workflows/ci-code.yml b/.github/workflows/ci-code.yml index 65fcc45860..d4517f2226 100644 --- a/.github/workflows/ci-code.yml +++ b/.github/workflows/ci-code.yml @@ -89,11 +89,26 @@ jobs: sudo apt install postgresql graphviz - name: Upgrade pip and setuptools - # It is crucial to update `setuptools` or the installation of `pymatgen` can break run: | - pip install --upgrade pip setuptools + pip install --upgrade pip pip --version + - name: Build pymatgen with compatible numpy + run: | + # This step is necessary because certain versions of `pymatgen` will not specify an explicit version of + # `numpy` in its build requirements, and so the latest version will be used. This causes problems, + # however, because this means that the compiled version of `pymatgen` can only be used with that version + # of `numpy` or higher, since `numpy` only guarantees forward compatibility of the ABI. If we want to + # run with an older version of `numpy`, we need to ensure that `pymatgen` is built with that same + # version. This we can accomplish by installing the desired version of `numpy` manually and then calling + # the install command for `pymatgen` with the `--no-build-isolation` flag. This flag will ensure that + # build dependencies are ignored and won't be installed (preventing the most recent version of `numpy` + # to be installed) and the build relies on those requirements already being present in the environment. + # We also need to install `wheel` because otherwise the `pymatgen` build will fail because `bdist_wheel` + # will not be available. + pip install numpy==1.21.4 wheel + pip install pymatgen==2022.0.16 --no-cache-dir --no-build-isolation + - name: Install aiida-core run: | pip install --use-feature=2020-resolver -r requirements/requirements-py-${{ matrix.python-version }}.txt diff --git a/.github/workflows/rabbitmq.yml b/.github/workflows/rabbitmq.yml index d868ef1629..a3fc311988 100644 --- a/.github/workflows/rabbitmq.yml +++ b/.github/workflows/rabbitmq.yml @@ -56,6 +56,22 @@ jobs: pip install --upgrade pip pip --version + - name: Build pymatgen with compatible numpy + run: | + # This step is necessary because certain versions of `pymatgen` will not specify an explicit version of + # `numpy` in its build requirements, and so the latest version will be used. This causes problems, + # however, because this means that the compiled version of `pymatgen` can only be used with that version + # of `numpy` or higher, since `numpy` only guarantees forward compatibility of the ABI. If we want to + # run with an older version of `numpy`, we need to ensure that `pymatgen` is built with that same + # version. This we can accomplish by installing the desired version of `numpy` manually and then calling + # the install command for `pymatgen` with the `--no-build-isolation` flag. This flag will ensure that + # build dependencies are ignored and won't be installed (preventing the most recent version of `numpy` + # to be installed) and the build relies on those requirements already being present in the environment. + # We also need to install `wheel` because otherwise the `pymatgen` build will fail because `bdist_wheel` + # will not be available. + pip install numpy==1.21.4 wheel + pip install pymatgen==2022.0.16 --no-cache-dir --no-build-isolation + - name: Install aiida-core run: | pip install -r requirements/requirements-py-3.8.txt diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 1c95ff1307..05555ef5f0 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -82,9 +82,30 @@ jobs: run: | sudo apt update sudo apt install postgresql graphviz + + - name: Upgrade pip + run: | + pip install --upgrade pip + pip --version + + - name: Build pymatgen with compatible numpy + run: | + # This step is necessary because certain versions of `pymatgen` will not specify an explicit version of + # `numpy` in its build requirements, and so the latest version will be used. This causes problems, + # however, because this means that the compiled version of `pymatgen` can only be used with that version + # of `numpy` or higher, since `numpy` only guarantees forward compatibility of the ABI. If we want to + # run with an older version of `numpy`, we need to ensure that `pymatgen` is built with that same + # version. This we can accomplish by installing the desired version of `numpy` manually and then calling + # the install command for `pymatgen` with the `--no-build-isolation` flag. This flag will ensure that + # build dependencies are ignored and won't be installed (preventing the most recent version of `numpy` + # to be installed) and the build relies on those requirements already being present in the environment. + # We also need to install `wheel` because otherwise the `pymatgen` build will fail because `bdist_wheel` + # will not be available. + pip install numpy==1.21.4 wheel + pip install pymatgen==2022.0.16 --no-cache-dir --no-build-isolation + - name: Install aiida-core run: | - pip install --upgrade pip setuptools pip install -r requirements/requirements-py-3.8.txt pip install --no-deps -e . - name: Run sub-set of test suite diff --git a/requirements/requirements-py-3.10.txt b/requirements/requirements-py-3.10.txt index 26607edf12..3c507d6f36 100644 --- a/requirements/requirements-py-3.10.txt +++ b/requirements/requirements-py-3.10.txt @@ -76,7 +76,7 @@ nbformat==5.1.3 nest-asyncio==1.4.3 networkx==2.6.3 notebook==6.4.5 -numpy==1.21.4 +numpy==1.22.0 packaging==21.3 palettable==3.3.0 pamqp==2.3.0 @@ -104,7 +104,7 @@ PyCifRW==4.4.3 pycparser==2.21 pydata-sphinx-theme==0.6.3 Pygments==2.10.0 -pymatgen==2022.0.16 +pymatgen==2022.1.9 Pympler==0.9 PyMySQL==0.9.3 PyNaCl==1.4.0 diff --git a/setup.json b/setup.json index 69fb3da726..b4861c103f 100644 --- a/setup.json +++ b/setup.json @@ -86,7 +86,7 @@ "PyCifRW~=4.4", "ase~=3.18", "matplotlib~=3.3,>=3.3.4", - "pymatgen>=2019.7.2,<=2022.02.03,!=2019.9.7", + "pymatgen>=2019.7.2,<=2022.1.9,!=2019.9.7", "pymysql~=0.9.3", "seekpath~=1.9,>=1.9.3", "spglib~=1.14"