Cannot resolve dependencies in "all" setup extra

[alexander-held]

Summary

Installing the "all" setup extra (which I assume is the replacement for "complete", see #2110) currently results in pip struggling to find compatible versions of dependencies.

OS / Environment

python:3.9 via Docker

Steps to Reproduce

> docker run -it --rm python:3.9 /bin/bash
$ pip install pyhf[all]@git+https://github.com/scikit-hep/pyhf.git

File Upload (optional)

No response

Expected Results

I expect the resolution of compatible dependencies to happen reasonably quickly.

Actual Results

pip keeps downgrading numpy/scipy versions.

pyhf Version

main@2b168b7

Code of Conduct

• I agree to follow the Code of Conduct

[matthewfeickert]

> docker run -it --rm python:3.9 /bin/bash
$ pip install pyhf[all]@git+https://github.com/scikit-hep/pyhf.git

@alexander-held the syntax

python -m pip install --upgrade "git+https://github.com/scikit-hep/pyhf.git#egg=pyhf[all]"

following the patterns described in https://pyhf.readthedocs.io/en/v0.7.0/installation.html#install-latest-development-version-from-github works.

Is package-name[extra]@git+github-url supposed to be the more preferable syntax? I have to admit I haven't seen that before.

[alexander-held]

I found that syntax while looking for a way to do this as [...]pyhf.git[all] did not work and I did not remember the #egg. I cannot find the exact place I saw it earlier, but one example is pypa/pip#6598 (comment). No idea whether they differ in any way, but using

python -m pip install --upgrade "git+https://github.com/scikit-hep/pyhf.git#egg=pyhf[all]"

similarly gets me into a downgrade loop in python:3.9 that is eventually broken by a lack of disk space in the container.

[matthewfeickert]

I cannot find the exact place I saw it earlier, but one example is pypa/pip#6598 (comment).

Oh, I see that

$ python -m pip install --upgrade "git+https://github.com/scikit-hep/pyhf.git#egg=pyhf[all]"

prints out

DEPRECATION: git+https://github.com/scikit-hep/pyhf.git#egg=pyhf[all] contains an egg fragment with a non-PEP 508 name pip 25.0 will enforce this behaviour change. A possible replacement is to use the req @ url syntax, and remove the egg fragment. Discussion can be found at pypa/pip#11617

@alexander-held oh you're on a M1 mac though, right? So inside your container is running under the linux/arm64 platform and so uname -m shows aarch64.

$ docker run --rm -ti --platform linux/arm64 python:3.9 /bin/bash
root@eab9f65bbf86:/# uname -m
aarch64

I have no idea what the support for all the packages that we depend on is for ARM64 macs at the moment. This might need to get added to the docs as a warning. :/

The following does work on an x86_64 machine:

python -m pip install --upgrade 'pyhf[all]@git+https://github.com/scikit-hep/pyhf.git'

and this new syntax should get added to the docs.

[matthewfeickert]

@alexander-held As

$ docker run --rm -ti --platform linux/arm64 python:3.9 /bin/bash
root@eab9f65bbf86:/# uname -m
aarch64
root@eab9f65bbf86:/# python -m venv venv && . venv/bin/activate
(venv) root@eab9f65bbf86:/# python -m pip --quiet install --upgrade pip setuptools wheel
(venv) root@eab9f65bbf86:/# python -m pip --quiet install scipy
(venv) root@eab9f65bbf86:/# python -m pip list
Package    Version
---------- -------
numpy      1.24.2
pip        23.0
scipy      1.10.0
setuptools 67.2.0
wheel      0.38.4
(venv) root@eab9f65bbf86:/#

works can you verify your problem both in a local virtual environment on your M1 mac and in a Python 3.10 Docker container? Though even these are slightly different as the aarch64 support for Docker is not the same as the macosx_*_arm64 support.

[alexander-held]

This is on M1, yes. Same behavior in a virtual environment and in python:3.10. I thought I could avoid any M1-specific behavior by going through the container but realize now that I cannot.

[matthewfeickert]

I thought I could avoid any M1-specific behavior by going through the container but realize now that I cannot.

Yeah, sadly this is why I've spent time learning how to cross compile things across both Docker's linux/amd64 and linux/arm64 platforms (c.f. matthewfeickert/pythia-python#31).

It would be useful to figure out which of the dependencies isn't providing an aarch64 or macosx_*_arm64 wheel. @alexander-held could you try to build a lock file locally on your M1 with

$ python -m pip install pip-tools
$ echo "pyhf[complete]" > requirements.in
$ pip-compile --generate-hashes --output-file requirements-arm.lock requirements.in

This is obviously using information from PyPI instead of GitHub, but I think if the install is failing on your M1 then the lock file will fail as well. If not, then we should be able to understand what the constraints on SciPy are coming from.

[alexander-held]

Running that in Python 3.9:

$ pip-compile --generate-hashes --output-file requirements-arm.lock requirements.in
WARNING: the legacy dependency resolver is deprecated and will be removed in future versions of pip-tools. The default resolver will be changed to 'backtracking' in pip-tools 7.0.0. Specify --resolver=backtracking to silence this warning.
Using legacy resolver. Consider using backtracking resolver with `--resolver=backtracking`.
Could not find a version that matches tensorflow>=2.7.0 (from pyhf[complete]==0.7.0->-r requirements.in (line 1))
No versions found
Was https://pypi.org/simple reachable?

[matthewfeickert]

Thanks @alexander-held. Yeah, this is sadly because no TensorFlow release supports macosx_*_arm64 wheels yet (c.f. https://pypi.org/project/tensorflow/2.11.0/#files and tensorflow/tensorflow#57185). :(

They do have aarch64 wheels though, so can you rerun that in a python:3.10 Docker container and see what you get? (I'm asking you to do it as to run with --platform linux/arm64 with Docker requires runtime emulation on my x86 laptop which is very slow).

[alexander-held]

With python:3.10:

WARNING: the legacy dependency resolver is deprecated and will be removed in future versions of pip-tools. The default resolver will be changed to 'backtracking' in pip-tools 7.0.0. Specify --resolver=backtracking to silence this warning.
Using legacy resolver. Consider using backtracking resolver with `--resolver=backtracking`.
Could not find a version that matches jaxlib!=0.1.68,>=0.1.61 (from pyhf[complete]==0.7.0->-r requirements.in (line 1))
No versions found
Was https://pypi.org/simple reachable?

[matthewfeickert]

With python:3.10:

WARNING: the legacy dependency resolver is deprecated and will be removed in future versions of pip-tools. The default resolver will be changed to 'backtracking' in pip-tools 7.0.0. Specify --resolver=backtracking to silence this warning.
Using legacy resolver. Consider using backtracking resolver with `--resolver=backtracking`.
Could not find a version that matches jaxlib!=0.1.68,>=0.1.61 (from pyhf[complete]==0.7.0->-r requirements.in (line 1))
No versions found
Was https://pypi.org/simple reachable?

Ah @alexander-held we're again getting bit by the difference in support between aarch64 and macosx_*_arm64 wheels. tensorflow supports aarch64 through tensorflow-aarch64 and tensorflow and so is installable in linux/arm64 Docker but not a M2 Mac, but jaxlib only supports macosx_*_arm64 and so is installable on a M2 Mac but not linux/arm64 Docker. So at the moment there is no extra that is able to install all of the backends simultaneously from PyPI — sorry, that's a frustrating answer.

You might try a micromamba environment though as there are osx-arm64 compatible releases of jaxlib, tensorflow, and pytorch on conda-forge currently.

[alexander-held]

There's a more fundamental question here: can the dependency on all the backends in conftest.py be relaxed? For a specific example, #2109 does not care at all about backends but to run any test one needs all the backends since the config otherwise fails. Not sure of the exact mechanism though in which they get pulled in and how feasible it is to relax this.

[matthewfeickert]

There's a more fundamental question here: can the dependency on all the backends in conftest.py be relaxed?

I would agree with you. We have Issue #1454 for this, but we haven't worked on it seriously. As @kratsg wrote the original setup for conftest.py I'm not sure if he has thoughts on how we might separate things out to allow for testing a subset of backends, but if he does it would be great to get thoughts in Issue #1454.