numpy 1.19.2

[jhein32]

Hi,

I was trying to build silx-0.13.2 via a pip install. This pulls in fabio 0.10.2 as a dependency. Building fabio fails:

ERROR: fabio 0.10.2 has requirement numpy>=1.19.2, but you'll have numpy 1.17.3 which is incompatible.

numpy 1.19.2 was only released on 10th September. fabio 0.10.2 is from May. This doesn't make sense to me. Could this please be looked at? I hope I am not overlooking something.

[kif]

Thomas finished the release of silx yesterday. Now binary wheels are available for most architectures. Which one are your using ?

lintaillefer:/tmp % python3 -m venv debug
lintaillefer:/tmp % source debug/bin/activate
(debug) lintaillefer:/tmp % pip install pip --upgrade
Cache entry deserialization failed, entry ignored
Collecting pip
  Using cached https://files.pythonhosted.org/packages/4e/5f/528232275f6509b1fff703c9280e58951a81abe24640905de621c9f81839/pip-20.2.3-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 18.1
    Uninstalling pip-18.1:
      Successfully uninstalled pip-18.1
Successfully installed pip-20.2.3
(debug) lintaillefer:/tmp % pip install silx
Collecting silx
  Downloading silx-0.13.2-cp37-cp37m-manylinux2014_x86_64.whl (12.9 MB)
     |████████████████████████████████| 12.9 MB 3.5 MB/s 
Collecting six
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Requirement already satisfied: setuptools in ./debug/lib/python3.7/site-packages (from silx) (40.8.0)
Collecting h5py
  Using cached h5py-2.10.0-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Collecting fabio>=0.9
  Downloading fabio-0.10.2-cp37-cp37m-manylinux2014_x86_64.whl (2.3 MB)
     |████████████████████████████████| 2.3 MB 35.7 MB/s 
Collecting numpy>=1.14.5
  Downloading numpy-1.19.2-cp37-cp37m-manylinux2010_x86_64.whl (14.5 MB)
     |████████████████████████████████| 14.5 MB 32.8 MB/s 
Installing collected packages: six, numpy, h5py, fabio, silx
Successfully installed fabio-0.10.2 h5py-2.10.0 numpy-1.19.2 silx-0.13.2 six-1.15.0

[kif]

If you want to rebuild fabio from sources, you should install cython to re-generate the C-code which may depend on the version of numpy used.

[jhein32]

Hi,

thanks for getting back to me so quickly. To isolate issue I just tried an fabio installation via pip. Due to interoperability requirements we us a python, numpy and h5py installation which we have on the cluster. As mentioned this comes with numpy 1.17.3. The pip now tries to upgrade the numpy to 1.19.2:

-bash-4.2$ pip install --install-option="--prefix=$HOME/fabio_test" fabio
/fast/sw/easybuild/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages/pip-19.2.3-py3.7.egg/pip/_internal/commands/install.py:243: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
  cmdoptions.check_install_build_global(options)
Collecting fabio
  Using cached https://files.pythonhosted.org/packages/88/56/970f1c13fbb65d8906c0d3aa13113d6d8b51ac1310a796f5c583a0d80076/fabio-0.10.2.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: setuptools in /fast/sw/easybuild/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from fabio) (41.2.0)
Collecting numpy>=1.19.2 (from fabio)
  Using cached https://files.pythonhosted.org/packages/bf/e8/15aea783ea72e2d4e51e3ec365e8dc4a1a32c9e5eb3a6d695b0d58e67cdd/numpy-1.19.2.zip
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Skipping bdist_wheel for fabio, due to binaries being disabled for it.
Skipping bdist_wheel for numpy, due to binaries being disabled for it.
Installing collected packages: numpy, fabio
  Found existing installation: numpy 1.17.3
    Uninstalling numpy-1.17.3:
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: 'f2py3.7'
Consider using the `--user` option or check the permissions.

-bash-4.2$ ls /fast/sw/easybuild/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages/

Based on what I have seen before, it seems that fabio prescribes the latest 1.19.2 version of numpy (which was released only a few days ago), which for an installation in a multi user HPC system makes matters really difficult.

When I tried silx yesterday, that accepted the numpy 1.17.3 (which is less than a year old).

[jhein32]

Following my previous input, I tried to build fabio 0.9.0, which is older than numpy 1.17.3. I get the same issue, that fabio is unhappy with numpy 1.17.3:

-bash-4.2$ pip install --install-option="--prefix=$HOME/fabio_test_=_9_0" fabio==0.9.0
/fast/sw/easybuild/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages/pip-19.2.3-py3.7.egg/pip/_internal/commands/install.py:243: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
  cmdoptions.check_install_build_global(options)
Collecting fabio==0.9.0
  Downloading https://files.pythonhosted.org/packages/c8/95/e995426701a9e3a4cb251367d8a02c696e7e2bff2b96b476c97b4b2c37e1/fabio-0.9.0.tar.gz (1.2MB)
     |████████████████████████████████| 1.2MB 3.7MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting numpy>=1.19.2 (from fabio==0.9.0)
  Using cached https://files.pythonhosted.org/packages/bf/e8/15aea783ea72e2d4e51e3ec365e8dc4a1a32c9e5eb3a6d695b0d58e67cdd/numpy-1.19.2.zip
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: setuptools in /fast/sw/easybuild/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from fabio==0.9.0) (41.2.0)
Skipping bdist_wheel for fabio, due to binaries being disabled for it.
Skipping bdist_wheel for numpy, due to binaries being disabled for it.
Installing collected packages: numpy, fabio
  Found existing installation: numpy 1.17.3
    Uninstalling numpy-1.17.3:
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: 'f2py3.7'
Consider using the `--user` option or check the permissions.

[kif]

Are you running on an exotic architecture ? Why does it want to build from scratch instead of using binary wheels ?
I suspect it is the --install-option which implies rebuilding the package.

For info, I am working with a numpy 1.16 with the development version of fabio. So the issue is not there. The minimum version for numpy should be 1.14.5.
Is there a bug in pip ? is it possible to upgrade pip ? Does your prefix contain a recent numpy version ?

[jhein32]

Hi,

Thank you so much for taking our issue so serious. Following your feedback, I tried using a different python installation with a different version of pip and got a warning, referencing issue: pypa/pip#7309

This might or might not be a smoking gun with regards what we are up to.

One of our requirements is relocation. It is a multi-user system and we can not just install things into the basic python installations and move the carpet under everyone's feet.

From a failed build (which was unfortunately allowed to write into the base python) I suspect fabio is not really listening to --install-option="--prefix=$HOME/fabio_test_=_9_0"

[kif]

One reasonable option would be to clone the virtual-environment you are using. I already used this tool with success:
https://github.com/edwardgeorge/virtualenv-clone

The use-case was similar to yours: provide users from a facility with their own venv already configured. The drawback is the size in the home directory (dozens of MB), but still one order of magnitude smaller than the same approach with conda.

[jhein32]

Hi, I have good news to you.

If I pip install --prefix=$HOME/fabio_test fabio, which is not using the --install-option it builds with the numpy and doesn't try to upgrade it to the latest numpy. I tested if I don't point it to my numpy it uses pip to install a numpy, which I take, that it is really using the numpy 1.17 which I provide.

So thanks for your help. Your comments were really helpful to point me in the right direction.

I think here is still an issue, why it doesn't accept the numpy 1.17 when forcing it to recompile fabio. As mentioned the silx is happy to rebuild with the numpy 1.17. I think I can now work around the issues. Let me know if you need more from me, or whether you decide to close this ticket.

Thanks again for your help.