Cannot install pymatgen==2019.7.2 with python3.6

[kpace]

Describe the bug
Cannot install pymatgen==2019.7.2 on ubuntu 16.04 in a clean virtualenv with python3.6.11.

To Reproduce

(test-venv) jenkins@ip:~/workspace/PR-212$ python --version
Python 3.6.11
(test-venv) jenkins@ip:~/workspace/PR-212$ pip --version
pip 20.3.1 from /usr/src/test-venv/lib/python3.6/site-packages/pip (python 3.6)
(test-venv) jenkins@ip-:~/workspace/PR-212$ pip3 install pymatgen==2019.7.2
Collecting pymatgen==2019.7.2
  Using cached pymatgen-2019.7.2.tar.gz (2.4 MB)
    ERROR: Command errored out with exit status 1:
     command: /usr/src/test-venv/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-gcjf5s0s/pymatgen_0d09ccb94359449695eafa5c38d65f5c/setup.py'"'"'; __file__='"'"'/tmp/pip-install-gcjf5s0s/pymatgen_0d09ccb94359449695eafa5c38d65f5c/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-98jjh4w1
         cwd: /tmp/pip-install-gcjf5s0s/pymatgen_0d09ccb94359449695eafa5c38d65f5c/
    Complete output (68 lines):
    Traceback (most recent call last):
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-exzgb8j1/numpy-1.20.0rc1/setup.py", line 30, in <module>
        Official docs: [http://pymatgen.org](http://pymatgen.org/)
    RuntimeError: Python version >= 3.7 required.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-gcjf5s0s/pymatgen_0d09ccb94359449695eafa5c38d65f5c/setup.py", line 164, in <module>
        'get_environment = pymatgen.cli.get_environment:main',
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/__init__.py", line 142, in setup
        _install_setup_requires(attrs)
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/__init__.py", line 137, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/dist.py", line 586, in fetch_build_eggs
        replace_conflicting=True,
      File "/usr/src/test-venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 780, in resolve
        replace_conflicting=replace_conflicting
      File "/usr/src/test-venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1063, in best_match
        return self.obtain(req, installer)
      File "/usr/src/test-venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1075, in obtain
        return installer(requirement)
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/dist.py", line 653, in fetch_build_egg
        return cmd.easy_install(req)
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 679, in easy_install
        return self.install_item(spec, dist.location, tmpdir, deps)
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 705, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 890, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 1158, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 1144, in run_setup
        run_setup(setup_script, args)
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/sandbox.py", line 253, in run_setup
        raise
      File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
        self.gen.throw(type, value, traceback)
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
        self.gen.throw(type, value, traceback)
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/sandbox.py", line 166, in save_modules
        saved_exc.resume()
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/sandbox.py", line 141, in resume
        six.reraise(type, exc, self._tb)
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/_vendor/six.py", line 685, in reraise
        raise value.with_traceback(tb)
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/usr/src/test-venv/lib/python3.6/site-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-exzgb8j1/numpy-1.20.0rc1/setup.py", line 30, in <module>
        Official docs: [http://pymatgen.org](http://pymatgen.org/)
    RuntimeError: Python version >= 3.7 required.
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
(test-venv) jenkins@ip:~/workspace/PR-212$

Desktop:

• OS: Ubuntu 16.04.5 LTS

Additional context
It seems it tries to download numpy-1.20.0rc1 which requires python >= 3.7. I also tried installing a lower version of numpy (1.15.1) and then installing pymatgen but it didn't work as well.

[shyuep]

I think this was a problem with past versions. Try the newest version of pymatgen instead. (Or if that fails, you should use a py37 environment. Py36 is pretty old and a lot of packages already dropped support for it)

[mkhorton]

Yes, to expand a little here, numpy has now officially dropped support for Python 3.6 in their latest version. While pymatgen should still work with 3.6 (all our tests still pass on 3.6) by using an older version of numpy such as 1.19.2, I concur with @shyuep that it would probably be easier to use Python 3.7+ if that's an option.

[csadorf]

Hi! I would appreciate if you could re-open the issue as the underlying bug breaks installation flows for all packages that depend on pymatgen, but still support Python 3.6. Concretely, this issue is currently causing a break of our standard installation path for AiiDA, see issue aiidateam/aiida-core#4614 and related pull request aiidateam/aiida-core#4615 . Important to mention that this issue has occurred in the past before, and was previously reported in #1520 .

I understand that the pymatgen maintainer perspective is to simply recommend the use of a newer Python version, but this is unfortunately not possible for some users and packages that have long-term support schedules. But even if that was an acceptable solution, this bug would still cause an issue, because it causes all constraints that dependent packages have placed on their numpy dependency to be ignored.

This problem would be much less severe if at least older versions of pymatgen would still behave properly, but the root cause makes it basically impossible to have any pymatgen dependency, because -- as was demonstrated by the OP -- even older versions of pymatgen will break. To put it plainly: unless the build setup is fixed, pymatgen will always cause issues when numpy drops support for older Python versions even for older versions of pymatgen, making it basically impossible to declare pymatgen as a dependency without giving up all control over numpy constraints.

[shyuep]

@csadorf I am unclear what is needed from pymatgen side. Certainly older versions of pymatgen will work for python 3.6. We do not specify a numpy version in the setup.py beyond ">=1.14.x". That means even relatively ancient numpy versions should work. The requirements.txt pinning for older versions of pymatgen are based on the numpy versions at those times and is in line with python best practices.

I did a test and even as recent as pymatgen 2020.8.3, install on py36 works.

conda create -n py36 python=3.6
conda activate py36
conda install numpy
pip install pymatgen==2020.8.3

gives me

Collecting pymatgen==2020.8.3
  Downloading pymatgen-2020.8.3-cp36-cp36m-macosx_10_14_x86_64.whl (3.3 MB)
     |████████████████████████████████| 3.3 MB 3.4 MB/s
Requirement already satisfied: numpy>=1.14.3 in ./opt/miniconda3/envs/py36/lib/python3.6/site-packages (from pymatgen==2020.8.3) (1.19.2)
Collecting dataclasses>=0.6
  Downloading dataclasses-0.8-py3-none-any.whl (19 kB)
Collecting matplotlib>=1.5
  Downloading matplotlib-3.3.3-cp36-cp36m-macosx_10_9_x86_64.whl (8.5 MB)
     |████████████████████████████████| 8.5 MB 29.1 MB/s
....

So it does seem the numpy setup.py is respected. I also tried it with your pymatgen==2019.7.2 and it works as well.

In short, I have no idea why your install is failing. I suspect it si an issue with your test-env caching. I suggest you use a clean environment and re-test.

Now, that said, you will have to pin to an older version of pymatgen. It seems anything before Sep 2020 should work fine. But newer features will not be available. Unfortunately, we cannot keep retaining compatibility with old versions. Once major packages such as numpy move on, newer versions of pymatgen will have to move on.

[sphuber]

Thanks for taking the time to investigate this @shyuep . The problem seems to be when installing pymatgen from the tarball distribution and so the wheel has to be built. In that case, something (I am not sure which package is responsible for this) is installing numpy==1.20.0rc1. Since this doesn't support Python 3.6, the build will crash. For an example, you can check this log of a run on Github Actions. Some of the relevant lines are:

Collecting pymatgen==2020.3.2
  Downloading pymatgen-2020.3.2.tar.gz (2.6 MB)
    ERROR: Command errored out with exit status 1:
     command: /opt/hostedtoolcache/Python/3.6.12/x64/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-s526ezws/pymatgen_06d20c37b3a749d187e77525e2902b22/setup.py'"'"'; __file__='"'"'/tmp/pip-install-s526ezws/pymatgen_06d20c37b3a749d187e77525e2902b22/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-1m_tqngz
         cwd: /tmp/pip-install-s526ezws/pymatgen_06d20c37b3a749d187e77525e2902b22/
    Complete output (68 lines):
    Traceback (most recent call last):
      File "/opt/hostedtoolcache/Python/3.6.12/x64/lib/python3.6/site-packages/setuptools/sandbox.py", line 154, in save_modules
        yield saved
      File "/opt/hostedtoolcache/Python/3.6.12/x64/lib/python3.6/site-packages/setuptools/sandbox.py", line 195, in setup_context
        yield
      File "/opt/hostedtoolcache/Python/3.6.12/x64/lib/python3.6/site-packages/setuptools/sandbox.py", line 250, in run_setup
        _execfile(setup_script, ns)
      File "/opt/hostedtoolcache/Python/3.6.12/x64/lib/python3.6/site-packages/setuptools/sandbox.py", line 45, in _execfile
        exec(code, globals, locals)
      File "/tmp/easy_install-q6kydy_z/numpy-1.20.0rc1/setup.py", line 30, in <module>
        if sys.platform.startswith('win') and platform.machine().endswith('64'):
    RuntimeError: Python version >= 3.7 required.

Now what is remarkable is the conditional which is included in the last line if sys.platform.startswith('win') and platform.machine().endswith('64'):. The tests are actually running on an Ubuntu 18.04 host. Now I am not sure if this is just a virtual machine that is running on a host runner that is actually using windows. Given that Github is now owned by MIcrosoft is not wholly unlikely.

I have been trying to reproduce this on my own machine which also runs Ubuntu 18.04, but without success. There, even in a clean Python 3.6 virtual environment, the pymatgen tarball is downloaded and built some problems. I am now wondering if the version of wheel or setuptools has anything to do with it. I am investigating this further and will update here.

[shyuep]

@sphuber I just retried on a Ubuntu server the exact same series of commands (the previous test was done on a Mac). There is still no error and if a numpy > 1.14 was installed, pymatgen installs without error. This was done on both a supercomputer as well as a virtual machine. So in short, I cannot reproduce the error you mention. I suspect it is something percuilar to the CI environment.

[mkhorton]

@shyuep did you pip install --upgrade pip, I was concerned maybe the new pip dependency resolver might be responsible (just a hunch)

[sphuber]

Note that if we pre-install a version of numpy that is compatible with the Python version of the environment, then the installation of pymatgen goes without problem (this is the current workaround that we use). So at least one condition for the problem to appear is for the environment to have no version of numpy installed whatsoever when pymatgen gets installed.

[sphuber]

@mkhorton we experienced the same problem over a year ago when Python 2.7 and 3.5 support was dropped by numpy and this was well before the new dependency resolver, so I don't think this is necessarily at fault.

[shyuep]

Interesting. Then I think this is an issue at either the versioning end or pip end. Basically, pip install should choose the latest version of numpy compatible with that python version (3.6 in your case). It is strange that it is attempting to install 1.20 which is clearly incompatible with py36.

[sphuber]

Interesting. Then I think this is an issue at either the versioning end or pip end. Basically, pip install should choose the latest version of numpy compatible with that python version (3.6 in your case). It is strange that it is attempting to install 1.20 which is clearly incompatible with py36.

I think that is exactly where the crux of the problem lies: if I understand things correctly it is not pip that installs numpy==1.20.0rc1. At the point when that occurs, I think it is another package manager or installation tool that interprets the requirements of pymatgen. My suspicion is that when the pymatgen wheel is getting built, it is either wheel or setuptools that is responsible for installing the dependencies and there the requirements imposed do not get respected.

If we look again at the failing build, it is the following command that fails:

/opt/hostedtoolcache/Python/3.6.12/x64/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-s526ezws/pymatgen_06d20c37b3a749d187e77525e2902b22/setup.py'"'"'; __file__='"'"'/tmp/pip-install-s526ezws/pymatgen_06d20c37b3a749d187e77525e2902b22/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-1m_tqngz

I have no idea who calls this command, but it seems likely that pip at this point is no longer involved. Since again setuptools is involved here as it gets imported, it seems to me the problem has to do with that.

[sphuber]

More evidence that setuptools is the culprit. I noticed that the minimum version you require is very low:

pymatgen/setup.py

Line 111 in bb6ca9e

setup_requires=['numpy>=1.14.3', 'setuptools>=18.0'],

I tried to update setuptools before installing pymatgen and that got rid of the problem. I guess the question now would be to figure out starting from which version of setuptools the problem was solved. If we find it, would it be a possible solution for you to increase the minimum required version of setuptools in your setup_requires? The current minimum version dates back to 2015.

[shyuep]

@sphuber I tried doing this from scratch in a clean ubuntu env and a clean conda, i.e., no numpy installed. I even let the system auto-find the correct version of pymatgen. I still cannot reproduce this error. This is the output.

(base) shyuep@ubuntu:~$ conda activate py36
(py36) shyuep@ubuntu:~$ pip install pymatgen
Collecting pymatgen
  Downloading pymatgen-2020.12.3.tar.gz (2.8 MB)
     |████████████████████████████████| 2.8 MB 3.8 MB/s
Collecting dataclasses>=0.6
  Downloading dataclasses-0.8-py3-none-any.whl (19 kB)
Collecting matplotlib>=1.5
  Downloading matplotlib-3.3.3-cp36-cp36m-manylinux1_x86_64.whl (11.6 MB)
     |████████████████████████████████| 11.6 MB 7.9 kB/s
Collecting cycler>=0.10
  Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.3.1-cp36-cp36m-manylinux1_x86_64.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 8.4 MB/s
Collecting monty>=3.0.2
  Downloading monty-4.0.2-py3-none-any.whl (62 kB)
     |████████████████████████████████| 62 kB 37 kB/s
Collecting networkx>=2.2
  Downloading networkx-2.5-py3-none-any.whl (1.6 MB)
     |████████████████████████████████| 1.6 MB 8.8 MB/s
Collecting decorator>=4.3.0
  Downloading decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting numpy>=1.14.3
  Using cached numpy-1.19.4-cp36-cp36m-manylinux2010_x86_64.whl (14.5 MB)
Collecting palettable>=3.1.1
  Downloading palettable-3.3.0-py2.py3-none-any.whl (111 kB)
     |████████████████████████████████| 111 kB 11.3 MB/s
Collecting pillow>=6.2.0
  Downloading Pillow-8.0.1-cp36-cp36m-manylinux1_x86_64.whl (2.2 MB)
     |████████████████████████████████| 2.2 MB 7.6 MB/s
Collecting plotly>=4.5.0
  Downloading plotly-4.14.1-py2.py3-none-any.whl (13.2 MB)
     |████████████████████████████████| 13.2 MB 56 kB/s
Collecting pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
     |████████████████████████████████| 67 kB 682 kB/s
Collecting python-dateutil>=2.1
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
     |████████████████████████████████| 227 kB 8.5 MB/s
Collecting retrying>=1.3.3
  Downloading retrying-1.3.3.tar.gz (10 kB)
Collecting ruamel.yaml>=0.15.6
  Downloading ruamel.yaml-0.16.12-py2.py3-none-any.whl (111 kB)
     |████████████████████████████████| 111 kB 8.8 MB/s
Collecting ruamel.yaml.clib>=0.1.2
  Downloading ruamel.yaml.clib-0.2.2-cp36-cp36m-manylinux1_x86_64.whl (549 kB)
     |████████████████████████████████| 549 kB 9.7 MB/s
Collecting scipy>=1.5.0
  Downloading scipy-1.5.4-cp36-cp36m-manylinux1_x86_64.whl (25.9 MB)
     |████████████████████████████████| 25.9 MB 1.3 MB/s
...

The bottom line is that it is successfully installed and numpy 1.19.4 (which is compatible with py36) was automatically installed.

One other thing you might want to look into is whether any existing packages in your env might be causing problems. Usually it is best to have the virtual env be isolated and not use any site-packages.

[shyuep]

@sphuber Thanks. Ok. I will make setuptools>=50 as the requirement. That should fix some of this for now. In general though, I think these things typically are updated by themselves....

[sphuber]

One other thing you might want to look into is whether any existing packages in your env might be causing problems. Usually it is best to have the virtual env be isolated and not use any site-packages.

We do use virtual environments. This is what we recommend to all users for installation and is definitely what we use in our CI. Each run gets a completely new and clean virtual environment. As described above, I really think it is due to the outdated version of setuptools. I suspect that in all your tests you had a more recent version where the problem was fixed.

[shyuep]

@sphuber Ok. I have increased the min setuptools version. That will not fix the old pymatgen releases, but hopefully it will prevent future releases from having issues.

[csadorf]

@shyuep Thank you very much for the help and sorry that I was not able to respond earlier. But I'm glad that @sphuber was available to clear up some of the questions. It seems that it is mostly figured out that setuptools is the culprit here.

Here is a minimal example that demonstrates that the bug was apparently resolved in setuptools version 42:

# fails:
$ docker run --env SETUPTOOLS_VERSION=41 --rm -it python:3.6  /bin/bash -c 'pip install -I setuptools==${SETUPTOOLS_VERSION} && pip install pymatgen==2019.7.2'
# succeeds:
$ docker run --env SETUPTOOLS_VERSION=42 --rm -it python:3.6  /bin/bash -c 'pip install -I setuptools==${SETUPTOOLS_VERSION} && pip install pymatgen==2019.7.2'

Same behavior when installed as dependency (example with aiida-core):

# fails:
 docker run --env SETUPTOOLS_VERSION=41 --rm -it python:3.6  /bin/bash -c 'pip install -I setuptools==${SETUPTOOLS_VERSION} && pip install aiida-core[atomic_tools]'
# succeeds:
 docker run --env SETUPTOOLS_VERSION=42 --rm -it python:3.6  /bin/bash -c 'pip install -I setuptools==${SETUPTOOLS_VERSION} && pip install aiida-core[atomic_tools]'

I therefore assume that it might be related to this change: https://setuptools.readthedocs.io/en/latest/history.html#id170

[ltalirz]

Sorry to add to a long thread, but while some of the issues may have been fixed, one root problem remains unaddressed:

pymatgen is using the setup_requires in the setup.py which is discouraged by setuptools.
The problem is that this results in using easy_install to fetch the build dependencies. easy_install is outdated and should not be used - it is significantly dumber than pip and can lead to all sorts of issues and incompatibilities [1].

The solution is to follow PEP-518 and use the pyproject.toml to specify the build dependencies.
I see that @shyuep just added a pyproject.toml an hour ago in a different context.
I will prepare a PR to use it also for the build dependencies.

[1] To take a recent example, our builds of the AiiDA documentation on readthedocs (using python 3.8) have been broken since December 4th due to an issue with numpy.ndarray size changed introduced by the pymatgen dependency - in this particular case I haven't checked that easy_install is the perpetrator but I strongly suspect it (we had a similar case before where it was).