install --root=... tries to remove previously globally installed version

[flying-sheep]

I tried to do pip install --root=... some-wheel-file.whl, which failed with a permission error because it tried to uninstall a globally installed version(!) and I luckily didn’t run with root permissions.

If i had, pip would have fucked up my package-manager-controlled global install.

What should happen is the same as with --user: install all dependencies to --root, then check if the package itself is installed in --root, remove it if so, and (re)install it to there. Leave it alone if it’s installed elsewhere.

Related to #3029

---

I can circumvent it by doing --root=... --ignore-installed --no-deps, but that is a hack:

What it does is ignoring installed packages to make it not try to remove previously installed versions (which it shouldn’t do in the first place if those versions don’t live in the specified --root!), and then I make it not install dependencies so that i end up with only that package installed in my --root.

[patatetom]

hi,

I think I still have this problem :-(

# pip install --upgrade pip
Requirement already up-to-date: pip in /usr/lib/python3.6/site-packages


# pip show pip
Name: pip
Version: 9.0.1
Summary: The PyPA recommended tool for installing Python packages.
Home-page: https://pip.pypa.io/
Author: The pip developers
Author-email: python-virtualenv@groups.google.com
License: MIT
Location: /usr/lib/python3.6/site-packages
Requires: 


# pip install --root /tmp/empty-dir/ python-evtx
Collecting python-evtx
  Using cached python-evtx-0.6.0.zip
Requirement already up-to-date: six in /usr/lib/python3.6/site-packages (from python-evtx)
Collecting pytest (from python-evtx)
  Using cached pytest-3.1.3-py2.py3-none-any.whl
Collecting hexdump (from python-evtx)
  Using cached hexdump-3.3.zip
Collecting pytest-cov (from python-evtx)
  Using cached pytest_cov-2.5.1-py2.py3-none-any.whl
Collecting py>=1.4.33 (from pytest->python-evtx)
  Using cached py-1.4.34-py2.py3-none-any.whl
Requirement already up-to-date: setuptools in /usr/lib/python3.6/site-packages (from pytest->python-evtx)
Collecting coverage>=3.7.1 (from pytest-cov->python-evtx)
  Using cached coverage-4.4.1-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: py, pytest, hexdump, coverage, pytest-cov, python-evtx
  Running setup.py install for hexdump ... done
Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python3.6/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/usr/lib/python3.6/site-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/usr/lib/python3.6/site-packages/pip/req/req_install.py", line 922, in install
    with open(inst_files_path, 'w') as f:
FileNotFoundError: [Errno 2] Aucun fichier ou dossier de ce type: 'usr/lib/python3.6/site-packages/hexdump-3.3-py3.6.egg-info/installed-files.txt'


# rm -rf /tmp/empty-dir


# pip install --verbose --root /tmp/empty-dir/ python-evtx
...
Installing collected packages: py, pytest, hexdump, coverage, pytest-cov, python-evtx


  changing mode of /tmp/empty-dir/usr/bin/py.test to 755
  changing mode of /tmp/empty-dir/usr/bin/pytest to 755
  Running setup.py install for hexdump ...     Running command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-qorm8d08/hexdump/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-19j23ih3-record/install-record.txt --single-version-externally-managed --root /tmp/empty-dir/ --compile
    running install
    running build
    running build_py
    creating build
    creating build/lib
    copying hexdump.py -> build/lib
    running install_lib
    copying build/lib/hexdump.py -> /tmp/empty-dir/usr/lib/python3.6/site-packages
    byte-compiling /tmp/empty-dir/usr/lib/python3.6/site-packages/hexdump.py to hexdump.cpython-36.pyc
    running install_data
    copying data/hexfile.bin -> /tmp/empty-dir/usr/lib/python3.6/site-packages/data
    running install_egg_info
    running egg_info
    creating hexdump.egg-info
    writing hexdump.egg-info/PKG-INFO
    writing dependency_links to hexdump.egg-info/dependency_links.txt
    writing top-level names to hexdump.egg-info/top_level.txt
    writing manifest file 'hexdump.egg-info/SOURCES.txt'
    reading manifest file 'hexdump.egg-info/SOURCES.txt'
    writing manifest file 'hexdump.egg-info/SOURCES.txt'
    removing '/tmp/empty-dir/usr/lib/python3.6/site-packages/hexdump-3.3-py3.6.egg-info' (and everything under it)
    Copying hexdump.egg-info to /tmp/empty-dir/usr/lib/python3.6/site-packages/hexdump-3.3-py3.6.egg-info
    running install_scripts
    writing list of installed files to '/tmp/pip-19j23ih3-record/install-record.txt'
done
Cleaning up...
  Removing source in /tmp/pip-build-qorm8d08/python-evtx
  Removing source in /tmp/pip-build-qorm8d08/hexdump
Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python3.6/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/usr/lib/python3.6/site-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/usr/lib/python3.6/site-packages/pip/req/req_install.py", line 922, in install
    with open(inst_files_path, 'w') as f:
FileNotFoundError: [Errno 2] Aucun fichier ou dossier de ce type: 'usr/lib/python3.6/site-packages/hexdump-3.3-py3.6.egg-info/installed-files.txt'

[pradyunsg]

Hey @yan12125! Thanks for filing this issue and sorry for the lack of response.

As I understand, you want --root to be treated as a simple change of directory where installation is done, with same behaviour for dealing with already installed packages. Is that correct?

[pradyunsg]

General advice: pip should not be run with sudo permissions since essentially there's remote code execution taking place.

[yan12125]

@pradyunsg

As I understand, you want --root to be treated as a simple change of directory where installation is done,

Yes. I need it for python -m ensurepip --root /xxx.

with same behaviour for dealing with already installed packages. Is that correct?

Not sure about how installed packages are handled, though.

[pradyunsg]

@yan12125 Does --prefix provide the needed behavior?

[yan12125]

Seems the result is similar for --prefix:

$ pip install --prefix=/home/yen/usr pip
Requirement already satisfied: pip in /home/yen/.local/lib/python3.6/site-packages (10.0.1)

pip shouldn't look for installed packages in default paths (e.g., ~/.local/lib/python3.6/site-packages) at all.

[cross]

I agree with @asfaltboy in #3029, --root should cause it to not consider things in sys.path locations outside of the specified root (or prefix). But, #3029 got closed shortly after that point in discussion as a dup of this one.

Now, I hit this myself 3 years later, ref #6355. Any chance this can get fixed/changed eventually?

[matthiasbeyer]

Experiencing the same issue.

[asfaltboy]

Now, 4 years after my original issue, I actually feel the behaviour as consistent. If you consider pip to be consistent with python, a package can be found by python, pip should also find it.

The workaround for "isolating packages to root" for both python and pip, would not be alter the behaviour of pip, but rather modify PYTHONPATH for example consider to only packages under root.

That said, regarding the --root option, it's not really clear how it is used "in the wild". If we can gather some feedback from users about how they use it, and why, then we can maybe improve it's interface to support additional use cases; e.g add some option to help users constraint PYTHONPATH.

I will speak for myself: if I look back at my the original use case, for which I opened a number of bugs under pypa projects, I was mostly trying to abuse packaging features to deploy applications (their dependencies) in some sort of isolation. I have since left that company, but the last thing I remember before we left, we got to a fully working deployment by installing wheels (from devpi) into a venv, and using entry_point scripts to run the apps. These days I just pip install dependencies globally within docker containers and don't bother packaging our applications.

[patatetom]

hi,
for my part, I use pip --root=... to install packages in a folder that serves as a basis for building a live system, a bit like a virtualenv...
these packages are not directly available with the package manager of my distribution.

[chrahunt]

Related to #4575, since defining an explicit scheme means we would use that same scheme for determining whether a package is installed in the first place.

[uiopaubo]

I found this bug from https://bugs.python.org/issue31916 which is filed in the context of DESTDIR.

When installing Python 3.7.6 and using "make install DESTDIR=...", pip gets installed as well. The problem is that it uses (in my case) /usr/local/bin/python3.7 as the interpreter, not something prefixed with the DESTDIR.

Interestingly, with Python 2.7.17, pip installed using ensurepip employs the correct interpreter. Perhaps this came about because the interpreter was already installed and the correct path identified.

In my case, I am inclined to think this is a Python 3 bug since the top-level Python Makefile uses the build Python to run ensurepip. This is a consequence of it appearing in the install rule. It can, however, be omitted by configuring with --with-ensurepip=no. (In Python 2, the default is not to run ensurepip.)

Once Python has been installed, the installed interpreter can then be run with "-m ensurepip" and either --default-pip or --altinstall to make the tool available. It will have the correct interpreter line in the script.

Sorry if this is not relevant to this particular bug, but I write this here for those following the link from the Python bug. I aim to re-open the Python bug in question.

[mensinda]

Any updates on this issue? I am also hitting this issue with make install DESTDIR.

[mawinter69]

I ran into a similar issue. My setup was
./configure prefix=/
make DESTDIR=/data/python-inst
And the ensurepip always claimed that the target directory is not writable and will install to the user which ended then in
<DESTDIR>/<HOME>/.local/...
It turned out that in the end the distutils does the following:

def change_root (new_root, pathname):
    """Return 'pathname' with 'new_root' prepended.  If 'pathname' is
    relative, this is equivalent to "os.path.join(new_root,pathname)".
    Otherwise, it requires making 'pathname' relative and then joining the
    two, which is tricky on DOS/Windows and Mac OS.
    """
    if os.name == 'posix':
        if not os.path.isabs(pathname):
            return os.path.join(new_root, pathname)
        else:
            return os.path.join(new_root, pathname[1:])

So in my case this method got pathname=//lib and the join will fail to prepend the new_root as pathname[1:] is still an absolute path
Solution for me was to use prefix=/path

[bioinfornatics]

I encounter the same issue as many others users.
I would like to know if they are a workaround ?
maybe I could to use PREFIX as DESTDIR + PREFIX but in such case I should fix RPATH isn't it ?

[jhgit]

As I understand the way the python build/install world, times are shifting:

https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html

Because of this pip issue, this leaves users to choose between potentially broken 'setup.py install --prefix' issues ("broken" due to various alleged problems with setuptools) and using 'pip install' plus hacky flags.

Are there some thoughts to a way forward to fix this? I'm not against the idea of poking around to come up with a candidate to fix this, but what has been investigated so far for this issue?

[septatrix]

hi, for my part, I use pip --root=... to install packages in a folder that serves as a basis for building a live system, a bit like a virtualenv... these packages are not directly available with the package manager of my distribution.

This is also my use case. Currently I use --ignore-installed instead though this comes at the disadvantage of overwriting system packages.

I have also tried to create a venv inside the image/chroot, however as python is running from outside the targets filesystem all the shebangs are messed up. Is there any way to customize the path of the interpreter where the shebangs point to?