easy_install installation location not controlled by --prefix

[adamjstewart]

Hi, Spack developer here.

One of our users noticed that when setuptools is installed, it tries to install its easy_install.py file to the site-packages directory of the Python installation, even though we explicitly provide an alternate --prefix to install to. This is particularly a problem for users who use a system Python with Spack that they don't have install permissions to. Is there some other way to tell easy_install.py where to install to besides the --prefix option of setup.py install? See spack/spack#13738 and spack/spack#13955 for the original issues.

@LDAmorim @ax3l

[psakievich]

I ran into this issue today. This is annoying bug. To get around it I had to do a new install of python and all my packages. It would be really great if this could be addressed soon.

[adamjstewart]

Are there any setuptools developers who can take a look at this issue? Note that "easy_install is deprecated" is not an answer. We aren't trying to use easy_install, we're trying to install setuptools, but easy_install is being installed to a system location.

[abravalheri]

Hi @adamjstewart, thanks for bringing this subject forward.

If I understood correctly you are trying to install setuptools itself, and not calling easy_install.

Could you provide a little snippet that shows how you are installing setuptools?

[adamjstewart]

Yes, see the spack-build-out.txt file in spack/spack#13738

[abravalheri]

I see... Thanks for that!
It seems that you guys are using python setup.py install.
I will try to have a look on that but I cannot promise I will manage it soon (unfortunately... sorry for that).

If you need something quick, I recommend having a look on https://installer.readthedocs.io/en/latest/.
Maybe you can write an auxiliary script and use it in your procedure (the maintainers are still trying to finalize a good design for its CLI, but you can borrow some ideas from https://github.com/pradyunsg/installer/pull/94 or https://github.com/pradyunsg/installer/pull/92).
It is highly likely that the standard install functionality in the Python ecosystem will rely on this package on the near future. Meanwhile you can also try https://github.com/layday/pyproject-install (transitional).

[adamjstewart]

Yes, we are planning on moving from python setup.py install to pip install in spack/spack#27798. @LDAmorim @ax3l can one of you see if this issue still exists when using spack/spack#27798?

I would love to use build and installer instead of pip, but unfortunately build/installer require build/installer to build and install all of their run-time dependencies 😆

[abravalheri]

Wow, I see the egg/chicken problem now!

I wonder if this problem with build, installer is already being discussed in https://github.com/pypa/packaging-problems or https://discuss.python.org/c/packaging/14...

[abravalheri]

Hi @adamjstewart, I tried to replicate (a simplified version of) this issue, using both a clone of the repository and the latest sdist in PyPI. You can follow the steps I took bellow:

git clone

cd /tmp
mkdir -p /tmp/_prefix
git clone https://github.com/pypa/setuptools
cd setuptools
python3 setup.py --no-user-cfg egg_info
python3 setup.py --no-user-cfg install --prefix=/tmp/_prefix --root=/

setuptools_clone_logs.txt

sdist

cd /tmp
wget https://files.pythonhosted.org/packages/d9/15/0c48951d6fe38d7b3396401f2cfc43c86bc32408ed6c637ce1a9c6df1bdd/setuptools-60.3.1.tar.gz
tar xf setuptools-60.3.1.tar.gz --directory=.
cd setuptools-60.3.1
rm -rf /tmp/_prefix
mkdir /tmp/_prefix
python3 setup.py --no-user-cfg install --prefix=/tmp/_prefix --root=/

setuptools_sdist_logs.txt

Conclusions

I cannot see in the logs any attempt to create site-packages/easy_install.py in a global folder, everything seems to be installed in the prefix as intended:

$ fd 'easy_install' /tmp/_prefix
/tmp/_prefix/lib/python3.8/site-packages/setuptools/command/__pycache__/easy_install.cpython-38.pyc
/tmp/_prefix/lib/python3.8/site-packages/setuptools/command/easy_install.py

Because of this I am assuming that:

1. either this issue was already fixed in a more updated version of setuptools,
2. or it is OSX specific.

If this issue is OSX specific, I am afraid I will not be able to help since I only have access to Ubuntu machines :/

Could you try again with the latest version of setuptools and see if this problem still exists?
Is there any chance the build system you guys are using is patching the installation schemes?

[FRidh]

@adamjstewart in Nixpkgs I've been experimenting with migrating from pip to build and installer. I strongly suggest to go that direction as well, in time. Currently we're stuck at requiring --prefix support. Could you also have a look at https://github.com/pradyunsg/installer/pull/94#issuecomment-1013876999. If I'm correct you need it as well.

Regarding this issue, here's what we do in Nixpkgs with setuptools and pip. Maybe it helps you. For bootstrapping setuptools we create an sdist first

${python.pythonForBuild.interpreter} setup.py egg_info
${python.pythonForBuild.interpreter} setup.py sdist --formats=gztar

# Here we untar the sdist and retar it in order to control the timestamps
# of all the files included
tar -xzf dist/${pname}-${version}.post0.tar.gz -C dist/
tar -czf dist/${name} -C dist/ --mtime="@$SOURCE_DATE_EPOCH" --sort=name ${pname}-${version}.post0

Then we install this sdist as well pip and wheel sources

# unpack ...

# Give folders a known name
mv pip* pip
mv setuptools* setuptools
mv wheel* wheel
# Set up PYTHONPATH. The above folders need to be on PYTHONPATH
# $out is where we are installing to and takes precedence
export PYTHONPATH="$out/${python.sitePackages}:$(pwd)/pip/src:$(pwd)/setuptools:$(pwd)/setuptools/pkg_resources:$(pwd)/wheel:$PYTHONPATH"

echo "Building setuptools wheel..."
pushd setuptools
rm pyproject.toml
${python.pythonForBuild.interpreter} -m pip install --no-build-isolation --no-index --prefix=$out  --ignore-installed --no-dependencies --no-cache .
popd

echo "Building wheel wheel..."
pushd wheel
${python.pythonForBuild.interpreter} -m pip install --no-build-isolation --no-index --prefix=$out  --ignore-installed --no-dependencies --no-cache .
popd

echo "Building pip wheel..."
pushd pip
${python.pythonForBuild.interpreter} -m pip install --no-build-isolation --no-index --prefix=$out  --ignore-installed --no-dependencies --no-cache .
popd

This results in a folder with a functioning pip + setuptools + wheel which can then be used for installing everything else.

---

And I now see you refer to it in your source 😄
spack/spack@48b8bfa#diff-77f66bf722abedd4cf4436a5a73c2dc9af4216fccaa34d6eb1f097b5ef3c0d89R33

[FRidh]

Wow, I see the egg/chicken problem now!

I wonder if this problem with build, installer is already being discussed in https://github.com/pypa/packaging-problems or https://discuss.python.org/c/packaging/14...

This is a common issue with bootstrapping. Unfortunately, there are quite some packages involved in this case. It has been discussed a bit on Discourse, and it's primarily up to the distro maintainers to work this out. For non-distro Python it is easy; just use pre-built wheels.

[adamjstewart]

Yes, we first used Nix's approach for bootstrapping pip/wheel/setuptools from source, but eventually moved to self-installing wheels as suggested in https://discuss.python.org/t/bootstrapping-a-specific-version-of-pip/12306. We found this to be much simpler since we need to support a wide range of Python versions that aren't all supported by the latest versions of pip/wheel/setuptools.

We would love to use build/installer, but unfortunately we need to support Python 2 for the foreseeable future. Many of our packages aren't compatible with Python 3, and are no longer maintained so there's no hope for them to be ported to Python 3. Many of these are important scientific libraries needed for reproducibility, so we can't just remove them.

[adamjstewart]

Getting back to the original issue, @LDAmorim @ax3l can you try to reproduce this with the tip of develop in Spack? If you can't reproduce the issue anymore then I'm guessing this has been fixed (either in setuptools or in Spack).

[abravalheri]

@adamjstewart @LDAmorim @ax3l, please also notice that using python -s might be problematic.
The current version of setuptools rely on a .pth file to load the correct version of distutils.
This is required during the transition period while distutils is deprecated and scheduled to be removed from the stdlib (i.e. will be necessary until the EoL of 3.11).