[BUG] pip install fails in Python 3.8 environment with unpinned or latest version

[oliverholworthy]

butDescribe the bug

pip install fails in a Python 3.8 environment with unpinned or latest version specified.

This started happening since the 23.06 release was published with an updated minimum version of Python 3.9 (#13196) - was previously Python 3.8.

It appears that the reason this error occurs is that the placeholder package published to the PyPI index doesn't contain the same minimum python version specifier (requires_python). So if cudf is installed with either no version specifier or with the 23.06 version, pip downloads the 23.6.0 package from PyPI, which is the 'latest' published version across both indexes {https://pypi.org/simple, https://pypi.nvidia.com} that 'supports' Python 3.8 according to the package requires_python configuration. And so raises the RuntimeError contained within the placeholder package.

Steps/Code to reproduce bug

With Python 3.8

• Fails with:
  • pip install cudf-cu11 --extra-index-url=https://pypi.nvidia.com (not expected)
  • pip install cudf-cu11==23.06 --extra-index-url=https://pypi.nvidia.com (expected, but not with the error below)
• Succeeds with:
  • pip install cudf-cu11==23.04 --extra-index-url=https://pypi.nvidia.com

> docker run -it python:3.8 pip install cudf-cu11 --extra-index-url=https://pypi.nvidia.com

Looking in indexes: https://pypi.org/simple, https://pypi.nvidia.com
Collecting cudf-cu11
  Downloading cudf-cu11-23.6.0.tar.gz (6.8 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [16 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-ui223aik/cudf-cu11_8f05f2529e5c4483b0203f5bb129e7ff/setup.py", line 137, in <module>
          raise RuntimeError(open("ERROR.txt", "r").read())
      RuntimeError:
      ###########################################################################################
      The package you are trying to install is only a placeholder project on PyPI.org repository.
      This package is hosted on NVIDIA Python Package Index.

      This package can be installed as:
      ```
      $ pip install --extra-index-url https://pypi.nvidia.com cudf-cu11
      ```
      ###########################################################################################

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

With Python 3.9 and 3.10

> docker run -it python:3.9 pip install cudf-cu11 --extra-index-url=https://pypi.nvidia.com

Looking in indexes: https://pypi.org/simple, https://pypi.nvidia.com
Collecting cudf-cu11
  Downloading https://pypi.nvidia.com/cudf-cu11/cudf_cu11-23.6.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (489.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 489.4/489.4 MB 8.4 MB/s eta 0:00:00
Collecting numpy>=1.21
  Downloading numpy-1.25.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.7/17.7 MB 53.0 MB/s eta 0:00:00
Collecting numba>=0.57
  Downloading numba-0.57.1-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 53.2 MB/s eta 0:00:00
Collecting pandas<1.6.0dev0,>=1.3
  Downloading pandas-1.5.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.2/12.2 MB 75.8 MB/s eta 0:00:00
Collecting nvtx>=0.2.1
  Downloading nvtx-0.2.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 441.3/441.3 kB 15.1 MB/s eta 0:00:00
...

Expected behavior

• pip install cudf-cu11 --extra-index-url=https://pypi.nvidia.com in Python 3.8 environment installs the last compatible version (23.04)
• pip install cudf-cu11==23.06 --extra-index-url=https://pypi.nvidia.com in Python 3.8 environment returns standard pip error saying Could not find a version that satisfies the requirement ...

[vyasr]

Thanks for reporting this! We're working on getting our placeholders updated to fix this for future releases.

[vyasr]

This issue was fixed. In newer versions, we use an alternate approach to the sdists on PyPI that fetch the right ones from our index, so any similar issues should be gone (although I'm sure other issues will manifest quite differently in the future 🙃).