Cannot install easybuild-easyconfigs using pip if setuptools>61

[ktaletsk]

Hi, I'm using easybuild in my Docker image, which uses miniconda environment for Python. Recently, after some updates I could not build my image anymore because pip install easybuild has failed with a large number of errors similar to package init file 'easybuild/__init__.py' not found (or not a regular file).

Full log

  19:19:53  Step 7/7 : RUN pip install --ignore-installed easybuild==4.5.1
19:19:53   ---> Running in c8da42b3c8a2
19:19:54  Collecting easybuild==4.5.1
19:19:54    Downloading easybuild-4.5.1.tar.gz (10 kB)
19:19:54    Preparing metadata (setup.py): started
19:19:55    Preparing metadata (setup.py): finished with status 'done'
19:19:55  Collecting easybuild-framework==4.5.1
19:19:55    Downloading easybuild-framework-4.5.1.tar.gz (2.0 MB)
19:19:55       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 40.4 MB/s eta 0:00:00
19:19:55    Preparing metadata (setup.py): started
19:19:57    Preparing metadata (setup.py): finished with status 'done'
19:19:57  Collecting easybuild-easyblocks==4.5.1
19:19:57    Downloading easybuild-easyblocks-4.5.1.tar.gz (512 kB)
19:19:57       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 512.9/512.9 KB 50.6 MB/s eta 0:00:00
19:19:57    Preparing metadata (setup.py): started
19:19:58    Preparing metadata (setup.py): finished with status 'done'
19:19:58  Collecting easybuild-easyconfigs==4.5.1
19:19:58    Downloading easybuild-easyconfigs-4.5.1.tar.gz (7.1 MB)
19:19:58       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.1/7.1 MB 87.0 MB/s eta 0:00:00
19:20:03    Preparing metadata (setup.py): started
19:20:05    Preparing metadata (setup.py): finished with status 'done'
19:20:05  Building wheels for collected packages: easybuild, easybuild-easyblocks, easybuild-easyconfigs, easybuild-framework
19:20:05    Building wheel for easybuild (setup.py): started
19:20:06    Building wheel for easybuild (setup.py): finished with status 'done'
19:20:06    Created wheel for easybuild: filename=easybuild-4.5.1-py3-none-any.whl size=10225 sha256=90c543a8f52f5748e7f39ce70c82c13ba856cd8598c47d967640644d04626eaa
19:20:06    Stored in directory: /home/jovyan/.cache/pip/wheels/e5/4d/03/71de5ed71b80ef1a238622c21c1d654cee06f80b32ba89681c
19:20:06    Building wheel for easybuild-easyblocks (setup.py): started
19:20:08    Building wheel for easybuild-easyblocks (setup.py): finished with status 'done'
19:20:08    Created wheel for easybuild-easyblocks: filename=easybuild_easyblocks-4.5.1-py3-none-any.whl size=784121 sha256=beecf56724cea8c7b4ed8cd061f440ab7d4d40cef3bcabab2c98af424cfba377
19:20:08    Stored in directory: /home/jovyan/.cache/pip/wheels/33/d1/f1/333e64b6bcbfa8c7735fd1065d429b1af23f87d45db763ef1c
19:20:08    Building wheel for easybuild-easyconfigs (setup.py): started
19:20:09    Building wheel for easybuild-easyconfigs (setup.py): finished with status 'error'
19:20:09  �[91m  error: subprocess-exited-with-error
19:20:09    
19:20:09    × python setup.py bdist_wheel did not run successfully.
19:20:09    │ exit code: 1
19:20:09    ╰─> [35 lines of output]
19:20:09        Installing version 4.5.1 (required versions: API >= 4, easyblocks >= 4.5)
19:20:09        running bdist_wheel
19:20:09        running build
19:20:09        running build_py
19:20:09        package init file 'easybuild/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/x/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/i/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/e/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/r/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/a/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/w/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/c/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/j/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/y/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/t/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/m/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/q/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/s/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/z/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/g/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/v/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/o/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/d/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/u/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/k/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/b/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/l/__init__.py' not found (or not a regular file)
19:20:09        package init file 'easybuild/easyconfigs/p/__init__.py' not found (or not a regular file)
19:20:09        creating build
19:20:09        creating build/lib
19:20:09        creating build/lib/easybuild
19:20:09        creating build/lib/easybuild/easyconfigs
19:20:09        creating build/lib/easybuild/easyconfigs/p
19:20:09        error: can't copy 'easybuild/easyconfigs/p/plotly.py': doesn't exist or not a regular file
19:20:09        [end of output]
19:20:09    
19:20:09    note: This error originates from a subprocess, and is likely not a problem with pip.
19:20:09  �[0m�[91m  ERROR: Failed building wheel for easybuild-easyconfigs
19:20:09  �[0m  Running setup.py clean for easybuild-easyconfigs
19:20:10    Building wheel for easybuild-framework (setup.py): started
19:20:13    Building wheel for easybuild-framework (setup.py): finished with status 'done'
19:20:13    Created wheel for easybuild-framework: filename=easybuild_framework-4.5.1-py3-none-any.whl size=3595163 sha256=23a751e990d25a9acebe225168d382f08eecaee45a3556d753c21fd30d5510bf
19:20:13    Stored in directory: /home/jovyan/.cache/pip/wheels/b4/f8/e1/4b87f276053032e2ddc1459b9270850d010f0875972bde7340
19:20:13  Successfully built easybuild easybuild-easyblocks easybuild-framework
19:20:13  Failed to build easybuild-easyconfigs
19:20:14  Installing collected packages: easybuild-framework, easybuild-easyconfigs, easybuild-easyblocks, easybuild
19:20:15    Running setup.py install for easybuild-easyconfigs: started
19:20:16    Running setup.py install for easybuild-easyconfigs: finished with status 'error'
19:20:16  �[91m  error: subprocess-exited-with-error
19:20:16    
19:20:16    × Running setup.py install for easybuild-easyconfigs did not run successfully.
19:20:16    │ exit code: 1
19:20:16    ╰─> [37 lines of output]
19:20:16        Installing version 4.5.1 (required versions: API >= 4, easyblocks >= 4.5)
19:20:16        running install
19:20:16        /opt/conda/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
19:20:16          warnings.warn(
19:20:16        running build
19:20:16        running build_py
19:20:16        package init file 'easybuild/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/x/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/i/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/e/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/r/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/a/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/w/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/c/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/j/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/y/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/t/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/m/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/q/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/s/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/z/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/g/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/v/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/o/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/d/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/u/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/k/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/b/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/l/__init__.py' not found (or not a regular file)
19:20:16        package init file 'easybuild/easyconfigs/p/__init__.py' not found (or not a regular file)
19:20:16        creating build
19:20:16        creating build/lib
19:20:16        creating build/lib/easybuild
19:20:16        creating build/lib/easybuild/easyconfigs
19:20:16        creating build/lib/easybuild/easyconfigs/p
19:20:16        error: can't copy 'easybuild/easyconfigs/p/plotly.py': doesn't exist or not a regular file
19:20:16        [end of output]
19:20:16    
19:20:16    note: This error originates from a subprocess, and is likely not a problem with pip.
19:20:16  �[0m�[91merror: legacy-install-failure
19:20:16  
19:20:16  × Encountered error while trying to install package.
19:20:16  ╰─> easybuild-easyconfigs
19:20:16  
19:20:16  note: This is an issue with the package mentioned above, not pip.
19:20:16  hint: See above for output from the failure.
19:20:17  �[0mThe command '/bin/sh -c pip install --ignore-installed easybuild==4.5.1' returned a non-zero code: 1

For a context, here are versions that I have in my container

	version
python	3.9.12
conda	4.10.3
pip	22.0.4
setuptools	61.2.0

I noticed on Stackoverflow someone recently had the same issue recently https://stackoverflow.com/questions/71648527/how-to-solve-easybuild-installation-problems

The issue seem to be isolated specifically in building a Python wheel for easybuild-easyconfigs package. I tried to isolate any issues related to my container by running popular public containers. Below are the results of my testing:

1. Python Docker image, docker run --rm -it python pip install easybuild-easyconfigs. Success.

	version
python	3.10.4
conda	N/A
pip	22.0.4
setuptools	58.1.0

Full log

Collecting easybuild-easyconfigs
Downloading easybuild-easyconfigs-4.5.3.tar.gz (7.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.3/7.3 MB 15.9 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Building wheels for collected packages: easybuild-easyconfigs
Building wheel for easybuild-easyconfigs (setup.py) ... done
Created wheel for easybuild-easyconfigs: filename=easybuild_easyconfigs-4.5.3-py3-none-any.whl size=22498617 sha256=c11bf548aea702685b75c08847304b62ea4702501d3e755cc7419dad60de945e
Stored in directory: /root/.cache/pip/wheels/38/81/59/0001c055d5ca749b81d1f0149dce8cd71d803e370ee4b3be4f
Successfully built easybuild-easyconfigs
Installing collected packages: easybuild-easyconfigs
Successfully installed easybuild-easyconfigs-4.5.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

3. Jupyter Minimal Notebook Docker image from March 21, docker run --rm -it jupyter/minimal-notebook:2022-03-21 pip install easybuild-easyconfigs==4.5.3. Success.

	version
python	3.9.10
conda	4.11.0
pip	22.0.3
setuptools	60.9.3

Full log

Collecting easybuild-easyconfigs==4.5.3
Downloading easybuild-easyconfigs-4.5.3.tar.gz (7.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.3/7.3 MB 16.8 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Building wheels for collected packages: easybuild-easyconfigs
Building wheel for easybuild-easyconfigs (setup.py) ... done
Created wheel for easybuild-easyconfigs: filename=easybuild_easyconfigs-4.5.3-py3-none-any.whl size=22498617 sha256=32a9aa13ca84779bd2a59f3c4ebd00995626c0a4b4fcaf830dd15b27d9dd5759
Stored in directory: /home/jovyan/.cache/pip/wheels/71/e6/f0/d353f5e1b8aff637cadf674d069ebf1e0a34e653f3ccded3ee
Successfully built easybuild-easyconfigs
Installing collected packages: easybuild-easyconfigs
Successfully installed easybuild-easyconfigs-4.5.3

4. Jupyter Minimal Notebook Docker image from March 28, docker run --rm -it jupyter/minimal-notebook:2022-03-28 pip install easybuild-easyconfigs==4.5.3. Failure.

	version
python	3.9.10
conda	4.12.0
pip	22.0.4
setuptools	61.1.1

Full log

Collecting easybuild-easyconfigs==4.5.3
Downloading easybuild-easyconfigs-4.5.3.tar.gz (7.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.3/7.3 MB 13.0 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Building wheels for collected packages: easybuild-easyconfigs
Building wheel for easybuild-easyconfigs (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
    Installing version 4.5.3 (required versions: API >= 4, easyblocks >= 4.5)
    running bdist_wheel
    running build
    running build_py
    package init file 'easybuild/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/v/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/a/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/g/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/k/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/z/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/i/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/__archive__/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/p/__init__.py' not found (or not a regular file)
    creating build
    creating build/lib
    creating build/lib/easybuild
    creating build/lib/easybuild/easyconfigs
    creating build/lib/easybuild/easyconfigs/p
    error: can't copy 'easybuild/easyconfigs/p/path.py': doesn't exist or not a regular file
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for easybuild-easyconfigs
Running setup.py clean for easybuild-easyconfigs
Failed to build easybuild-easyconfigs
Installing collected packages: easybuild-easyconfigs
Running setup.py install for easybuild-easyconfigs ... error
error: subprocess-exited-with-error

× Running setup.py install for easybuild-easyconfigs did not run successfully.
│ exit code: 1
╰─> [22 lines of output]
    Installing version 4.5.3 (required versions: API >= 4, easyblocks >= 4.5)
    running install
    /opt/conda/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    running build
    running build_py
    package init file 'easybuild/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/v/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/a/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/g/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/k/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/z/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/i/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/__archive__/__init__.py' not found (or not a regular file)
    package init file 'easybuild/easyconfigs/p/__init__.py' not found (or not a regular file)
    creating build
    creating build/lib
    creating build/lib/easybuild
    creating build/lib/easybuild/easyconfigs
    creating build/lib/easybuild/easyconfigs/p
    error: can't copy 'easybuild/easyconfigs/p/path.py': doesn't exist or not a regular file
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> easybuild-easyconfigs

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.  

I am confused what's going on. There seem to be no clear correlation of the issue with versions of Python, Conda or pip.

UPD: Added setuptools versions to the scenarios and it seems like the version 61 is the one causing issues

[ktaletsk]

Workaround for anyone affected -- build a wheel easybuild-easyconfigs in an environment where it works and copy it to your main environment. If using Docker, you can utilize a multistage build to do that

# Add this pre-build stage
----------
FROM python:3.9
RUN pip wheel easybuild-easyconfigs==4.5.3
----------

# Your normal Dockerfile
FROM ...


# Add this COPY command
----------
COPY --from=0 /easybuild* .
----------

RUN pip install easybuild_easyconfigs-4.5.3-py3-none-any.whl easybuild==4.5.3

After applying this changes, build proceeds without issues

[boegel]

Any relation to a similar issue that was reported at easybuilders/easybuild-framework#3984 ?

It seems like some kind of breaking change was introduced in a recent version of setuptools or pip that we're hitting?

[boegel]

@ktaletsk Can you update the issue description with the setuptools version that was used in each case?

[ktaletsk]

@boegel Added setuptools versions. Looks like easybuild-easyconfigs is incompatible with setuptools above version 61. As a check, downgrading setuptools to <61, and trying install again, works:

> docker run --rm -it jupyter/minimal-notebook:2022-03-28 bash
$ pip install easybuild-easyconfigs==4.5.3
# Same failure as above

# Downgrade setuptools
$ pip install --upgrade setuptools==60.9.3

# Try installing again
$ pip install easybuild-easyconfigs==4.5.3
Collecting easybuild-easyconfigs==4.5.3
  Using cached easybuild-easyconfigs-4.5.3.tar.gz (7.3 MB)
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: easybuild-easyconfigs
  Building wheel for easybuild-easyconfigs (setup.py) ... done
  Created wheel for easybuild-easyconfigs: filename=easybuild_easyconfigs-4.5.3-py3-none-any.whl size=22498617 sha256=97de381848c130dc53ad275c43c0b4a6e18c12200388072af10f9cc8410e791d
  Stored in directory: /home/jovyan/.cache/pip/wheels/71/e6/f0/d353f5e1b8aff637cadf674d069ebf1e0a34e653f3ccded3ee
Successfully built easybuild-easyconfigs
Installing collected packages: easybuild-easyconfigs
Successfully installed easybuild-easyconfigs-4.5.3

[boegel]

Thanks a lot, that's helpful.

It looks like we're being hit by a breaking change in setuptools 61.0, see https://github.com/pypa/setuptools/blob/main/CHANGES.rst#v6100, probably related to pypa/setuptools#2894...

The easyconfigs package is a special case, since we're only packaging up data files (easyconfigs + patch files, basically), rather than Python code.

It seems like we have some homework to do to ensure compatibility with future setuptools versions...

[boegel]

There's a trivial fix for this, see easybuilders/easybuild-easyconfigs#15206, we'll make sure that's included for the next EasyBuild release (but we also want to try setting up a CI workflow that catches problems like this)