Release 3.1.7 depends on numpy >= 1.22.0

[dihm]

Running a fresh install in a conda environment, I get the following error when instantiating an atom.

RuntimeError: module compiled against API version 0xf but this version of numpy is 0xe
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\naqsL\Miniconda3\envs\rydiqule\lib\site-packages\arc\alkali_atom_functions.py", line 208, in __init__
    from .arc_c_extensions import NumerovWavefunction
ImportError: numpy.core.multiarray failed to import

Much like #25, with a little digging, it appears the current release was built against numpy>=1.22.0, which included some more code reorganization. But that version of numpy hasn't made it to main conda channels yet. I did confirm that installing numpy 1.22.2 from pip removes the error.

[nikolasibalic]

Thank you for reporting this. It seems we run into this problem almost every year, so it was time... 😅

I think this is really issue with conda being slow with switching to latest versions. There is no easy way to know when this will occur and for how long it would make sense to keep ARC built fixed to older Numpy version. Also, if we do fix to older version we might not get possibly important Numpy bug fixes. So I think it's best solved with pip installation of latest version of Numpy as you did.

To try to save the trouble for other users, I opened a discussion #101 , where I linked three occurrences of this issue in the past.

[dihm]

So I think it's best solved with pip installation of latest version of Numpy as you did.

This is a totally fine response, and the explicit discussion page will hopefully help others find the quick solution.

I do have a caveat to this though. I'm currently working on a module that depends on numpy and ARC. When I went to pip numpy, I discovered that some core functionality I was using became significantly slower. This is apparently due to the fact that numpy on pip is built against OpenBLAS while numpy on conda is built against IntelMKL. Apparently I've been relying on some cleverness in the IntelMKL. This obviously isn't an ARC problem, but is a potential scenario where one has to install numpy from conda, even if it isn't the latest and greatest (yet).

But I wonder if there isn't some way for ARC to pull most recent available numpy (to keep on top of bug fixes), while still being backward compatible. Doing some quick searching, it appears numpy has chosen a backwards compatibility only model when it comes to build dependencies (so build with 1.16, run with 1.17 is fine; build with 1.17, run with 1.16 is not). In the numpy docs, they recommend building against the lowest version of numpy you want to support, and then pinning runtime dependencies a few versions into the future to protect against deprecation removals from breaking automated builds. Would this be an acceptable compromise?

[nikolasibalic]

Ah yes, I had run into that problem in the past too. Indeed Numpy backend is drastically different depending on the source of the distribution, which has strong influence on the performance.

conda-forge has newer versions compared to default channel, but in official Numpy docs they say it doesn't have good GPU support. On the other hand, maybe it's noteworthy comment in section "accelerated linear algebra libraries" where they say that conda-forge builds Numpy against dummy BLAS, and OpenBLAS or IntelMKL will be picked in runtime depending what user installs separately on the system. Might be useful if you ever need to combine latest versions but it's too involved for average users and package distribution.

Let's try what you suggested then. I limited now Numpy version to <=1.21.2 that as far as I see is supported in conda.
Could you check if this version (v3.1.8) works now for you well, or we need to re-adjust version range?

[dihm]

v3.1.8 works out of the box on conda. Thanks Nikola!

As for version ranges, I wonder if you could play with them a bit to allow users to install a newer version of numpy while building against an older one?

Forgive me if this is wrong since I'm fairly new to github workflows, but my understanding is that your build dependencies are specified in requirements.txt, but your runtime dependencies are defined separately in setup.py. By my reading of the numpy docs linked above, I think you could limit the numpy version you build against in requirements.txt, but allow the runtime dependency in setup.py to be unbounded on the top (allowing users to install and run against a newer version of numpy). Numpy guarantees that their API/ABI (at runtime) is backward compatible with builds against versions at least two minor releases back.

And if you want to make things a little easier on yourself, so you don't have to keep updating build pins as releases march forward, scipy provides a meta pip package for exactly this purpose called oldest-supported-numpy. Apparently this package should track and scale with python versions and platforms seemlessly. Fair warning, I haven't needed to do any of this kind of stuff before so your experience may be different. It is the recommended method by the numpy devs though.

In any case, if you want to attempt these build shenanigans, I'm happy to test. If you want to leave it as is, I'm obviously happy with that too. I'm just glad ARC exists and is such a great resource!

[nikolasibalic]

Many thanks for the feedback David! This is fantastic, since not a lot of people are willing to dive into built dependencies web, and even smaller number of people comes out with a good advice, as you did. 😄

I have implemented suggested changes, and it seems that things worked out as planned. 🤞 A new version is v3.1.9 . Let me know if you notice any problems in your tests!

[dihm]

That is very high praise coming from you Nikola. Glad I could help.

This works perfectly for me as well.