Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ModuleNotFoundError: No module named 'distutils.cmd' #124

Closed
BansheePrime opened this issue Aug 17, 2021 · 25 comments
Closed

ModuleNotFoundError: No module named 'distutils.cmd' #124

BansheePrime opened this issue Aug 17, 2021 · 25 comments

Comments

@BansheePrime
Copy link

pip installation on Debian 11 via get-pip.py
python 3.9.2

root@debi:~# python3 get-pip.py
Traceback (most recent call last):
  File "/root/get-pip.py", line 24556, in <module>
    main()
  File "/root/get-pip.py", line 139, in main
    bootstrap(tmpdir=tmpdir)
  File "/root/get-pip.py", line 115, in bootstrap
    monkeypatch_for_cert(tmpdir)
  File "/root/get-pip.py", line 96, in monkeypatch_for_cert
    from pip._internal.commands.install import InstallCommand
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpc1mpf262/pip.zip/pip/_internal/commands/__init__.py", line 9, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpc1mpf262/pip.zip/pip/_internal/cli/base_command.py", line 12, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpc1mpf262/pip.zip/pip/_internal/cli/cmdoptions.py", line 23, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpc1mpf262/pip.zip/pip/_internal/cli/parser.py", line 12, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpc1mpf262/pip.zip/pip/_internal/configuration.py", line 27, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpc1mpf262/pip.zip/pip/_internal/utils/misc.py", line 42, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpc1mpf262/pip.zip/pip/_internal/locations/__init__.py", line 14, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmpc1mpf262/pip.zip/pip/_internal/locations/_distutils.py", line 9, in <module>
ModuleNotFoundError: No module named 'distutils.cmd'
@theo-brown
Copy link

theo-brown commented Aug 18, 2021

Same here, also on Debian

@franklinscudder
Copy link

Also got this error on Debian 10 Buster, Python 3.7.3.

@PawelZ-RD
Copy link

Same on Ubuntu 20.04, Python 3.9.5
It used to work fine couple weeks ago...

@xavfernandez
Copy link
Member

It's likely because you're on a system that does not install the distutils module alongside python even though it is still part of the python standard library. sudo apt-get install python3-distutils (or something like that) should solve your issue.

@PawelZ-RD
Copy link

Sadly, on Ubuntu 20.04, python3-distutils is Python3.8 package. python3.9-distutils is a virtual that installs python3-distutils.
The reason I am trying to install pip via get-pip.py is to have Python3.9 only, with no 3.8.
I seems, I will have to get distutils directly from source first (what should be covered in get-pip.py IMHO).

Now, I see why it 'worked' before, it was Python 3.8

@PawelZ-RD
Copy link

PawelZ-RD commented Aug 24, 2021

After manually adding distutils:

# python3.9 get-pip.py
Collecting pip
  Downloading pip-21.2.4-py3-none-any.whl (1.6 MB)
     |████████████████████████████████| 1.6 MB 31.9 MB/s
Collecting setuptools
  Downloading setuptools-57.4.0-py3-none-any.whl (819 kB)
     |████████████████████████████████| 819 kB 37.0 MB/s
Collecting wheel
  Downloading wheel-0.37.0-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel, setuptools, pip
Successfully installed pip-21.2.4 setuptools-57.4.0 wheel-0.37.0

# pip --version
Traceback (most recent call last):
  File "/usr/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip'

# pip3 --version
Traceback (most recent call last):
  File "/usr/bin/pip3", line 5, in <module>
    from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip'

# ls -l /usr/lib/python3.9/site-packages/
total 36
drwxr-xr-x 3 root root 4096 Aug 24 16:40 _distutils_hack
-rw-r--r-- 1 root root  152 Aug 24 16:40 distutils-precedence.pth
drwxr-xr-x 5 root root 4096 Aug 24 16:40 pip
drwxr-xr-x 2 root root 4096 Aug 24 16:40 pip-21.2.4.dist-info
drwxr-xr-x 6 root root 4096 Aug 24 16:40 pkg_resources
drwxr-xr-x 7 root root 4096 Aug 24 16:40 setuptools
drwxr-xr-x 2 root root 4096 Aug 24 16:40 setuptools-57.4.0.dist-info
drwxr-xr-x 5 root root 4096 Aug 24 16:40 wheel
drwxr-xr-x 2 root root 4096 Aug 24 16:40 wheel-0.37.0.dist-info

This is caused by pip being installed in site-packages, while it is not included in default sys.path.

Could you update get-pip.py to install pip and its friends in either sys.prefix/lib/pythonX.Y/dist-packages or sys.prefix/lib/pythonX.Y so pip can be used without need to update default sys.path?
Thanks!

@xavfernandez
Copy link
Member

Could you update get-pip.py to install pip and its friends in either sys.prefix/lib/pythonX.Y/dist-packages or sys.prefix/lib/pythonX.Y so pip can be used without need to update default sys.path?

pip (which get-pip.py uses) installs itself in the directory advertised by the interpreter used to run it.
The logic is currently mainly here https://github.com/pypa/pip/blob/main/src/pip/_internal/locations/_distutils.py#L24-L102 and your interpreter is likely catching globally installed configuration files advertising /usr/lib/python3.9/site-packages/.

With pip moving to rely on sysconfig (for python 3.10 onwards) instead this should stop happening in a near future.

@xavfernandez
Copy link
Member

# pip --version
Traceback (most recent call last):
  File "/usr/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip'

# pip3 --version
Traceback (most recent call last):
  File "/usr/bin/pip3", line 5, in <module>
    from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip'

A better check would actually be # python3.9 -m pip --version to avoid the issue of "what interpreter will pip & pip3 scripts use ?".

@pradyunsg
Copy link
Member

In general, if you're outside of a Python virtual environment, you should invoke pip as python3.x -m pip.

@PawelZ-RD
Copy link

PawelZ-RD commented Aug 26, 2021

A better check would actually be # python3.9 -m pip --version to avoid the issue of "what interpreter will pip & pip3 scripts use ?".

# ls -l /usr/lib/python3.9/site-packages/
total 36
drwxr-xr-x 3 root root 4096 Aug 26 13:28 _distutils_hack
-rw-r--r-- 1 root root  152 Aug 26 13:28 distutils-precedence.pth
drwxr-xr-x 5 root root 4096 Aug 26 13:28 pip
drwxr-xr-x 2 root root 4096 Aug 26 13:28 pip-21.2.4.dist-info
drwxr-xr-x 6 root root 4096 Aug 26 13:28 pkg_resources
drwxr-xr-x 7 root root 4096 Aug 26 13:28 setuptools
drwxr-xr-x 2 root root 4096 Aug 26 13:28 setuptools-57.4.0.dist-info
drwxr-xr-x 5 root root 4096 Aug 26 13:28 wheel
drwxr-xr-x 2 root root 4096 Aug 26 13:28 wheel-0.37.0.dist-info

# python3.9 -m pip --version
/usr/bin/python3.9: No module named pip

# cd /usr/lib/python3.9/site-packages/
# python3.9 -m pip --version
pip 21.2.4 from /usr/lib/python3.9/site-packages/pip (python 3.9)

# python3.9 -c 'import sys; print(sys.path)'
['', '/usr/lib/python39.zip', '/usr/lib/python3.9', '/usr/lib/python3.9/lib-dynload', '/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages']

As I mentioned before, the issue is caused by pip being installed in location that is not in default sys.path
Since Python 3.10 is not there yet, and will not be adopted widely straight away, could you change get-pip.py to check for sys.path values and use one of paths it provides, please?

@xavfernandez
Copy link
Member

It would certainly make sense for pip to trigger a warning if it ends up installing packages outside of its sys.path 👍

Again, your interpreter seems to be misconfigured, advertising an installation scheme not included in its sys.path and I encourage you to fix this part :)

@xavfernandez
Copy link
Member

As additional options, you could try python3.9 get-pip.py --isolated or python3.9 get-pip.py --target <the directory where you want pip installed>.

@PawelZ-RD
Copy link

PawelZ-RD commented Aug 26, 2021

It would certainly make sense for pip to trigger a warning if it ends up installing packages outside of its sys.path 👍

That will be helpful! 👍

Again, your interpreter seems to be misconfigured, advertising an installation scheme not included in its sys.path and I encourage you to fix this part :)

It is what Ubuntu 20.04 package python3.9 delivers. Actually python3.8 does same thing:

# apt-get install -qq pyton3.8
E: Unable to locate package pyton3.8
E: Couldn't find any package by glob 'pyton3.8'
E: Couldn't find any package by regex 'pyton3.8'
root@871f1b979e29:/# apt-get install -y -qq python3.8
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libssl1.1:amd64.
(Reading database ... 4127 files and directories currently installed.)
Preparing to unpack .../00-libssl1.1_1.1.1f-1ubuntu2.8_amd64.deb ...
Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.8) ...
Selecting previously unselected package libpython3.8-minimal:amd64.
Preparing to unpack .../01-libpython3.8-minimal_3.8.10-0ubuntu1~20.04_amd64.deb ...
Unpacking libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04) ...
Selecting previously unselected package libexpat1:amd64.
Preparing to unpack .../02-libexpat1_2.2.9-1build1_amd64.deb ...
Unpacking libexpat1:amd64 (2.2.9-1build1) ...
Selecting previously unselected package python3.8-minimal.
Preparing to unpack .../03-python3.8-minimal_3.8.10-0ubuntu1~20.04_amd64.deb ...
Unpacking python3.8-minimal (3.8.10-0ubuntu1~20.04) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../04-libmagic-mgc_1%3a5.38-4_amd64.deb ...
Unpacking libmagic-mgc (1:5.38-4) ...
Selecting previously unselected package libmagic1:amd64.
Preparing to unpack .../05-libmagic1_1%3a5.38-4_amd64.deb ...
Unpacking libmagic1:amd64 (1:5.38-4) ...
Selecting previously unselected package file.
Preparing to unpack .../06-file_1%3a5.38-4_amd64.deb ...
Unpacking file (1:5.38-4) ...
Selecting previously unselected package libmpdec2:amd64.
Preparing to unpack .../07-libmpdec2_2.4.2-3_amd64.deb ...
Unpacking libmpdec2:amd64 (2.4.2-3) ...
Selecting previously unselected package mime-support.
Preparing to unpack .../08-mime-support_3.64ubuntu1_all.deb ...
Unpacking mime-support (3.64ubuntu1) ...
Selecting previously unselected package readline-common.
Preparing to unpack .../09-readline-common_8.0-4_all.deb ...
Unpacking readline-common (8.0-4) ...
Selecting previously unselected package libreadline8:amd64.
Preparing to unpack .../10-libreadline8_8.0-4_amd64.deb ...
Unpacking libreadline8:amd64 (8.0-4) ...
Selecting previously unselected package libsqlite3-0:amd64.
Preparing to unpack .../11-libsqlite3-0_3.31.1-4ubuntu0.2_amd64.deb ...
Unpacking libsqlite3-0:amd64 (3.31.1-4ubuntu0.2) ...
Selecting previously unselected package libpython3.8-stdlib:amd64.
Preparing to unpack .../12-libpython3.8-stdlib_3.8.10-0ubuntu1~20.04_amd64.deb ...
Unpacking libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04) ...
Selecting previously unselected package python3.8.
Preparing to unpack .../13-python3.8_3.8.10-0ubuntu1~20.04_amd64.deb ...
Unpacking python3.8 (3.8.10-0ubuntu1~20.04) ...
Selecting previously unselected package xz-utils.
Preparing to unpack .../14-xz-utils_5.2.4-1ubuntu1_amd64.deb ...
Unpacking xz-utils (5.2.4-1ubuntu1) ...
Setting up libexpat1:amd64 (2.2.9-1build1) ...
Setting up mime-support (3.64ubuntu1) ...
Setting up libmagic-mgc (1:5.38-4) ...
Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.8) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.30.0 /usr/local/share/perl/5.30.0 /usr/lib/x86_64-linux-gnu/perl5/5.30 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.30 /usr/share/perl/5.30 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Setting up libsqlite3-0:amd64 (3.31.1-4ubuntu0.2) ...
Setting up libmagic1:amd64 (1:5.38-4) ...
Setting up file (1:5.38-4) ...
Setting up xz-utils (5.2.4-1ubuntu1) ...
update-alternatives: using /usr/bin/xz to provide /usr/bin/lzma (lzma) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/lzma.1.gz because associated file /usr/share/man/man1/xz.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/unlzma.1.gz because associated file /usr/share/man/man1/unxz.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzcat.1.gz because associated file /usr/share/man/man1/xzcat.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzmore.1.gz because associated file /usr/share/man/man1/xzmore.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzless.1.gz because associated file /usr/share/man/man1/xzless.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzdiff.1.gz because associated file /usr/share/man/man1/xzdiff.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzcmp.1.gz because associated file /usr/share/man/man1/xzcmp.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzgrep.1.gz because associated file /usr/share/man/man1/xzgrep.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzegrep.1.gz because associated file /usr/share/man/man1/xzegrep.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzfgrep.1.gz because associated file /usr/share/man/man1/xzfgrep.1.gz (of link group lzma) doesn't exist
Setting up libmpdec2:amd64 (2.4.2-3) ...
Setting up readline-common (8.0-4) ...
Setting up libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04) ...
Setting up libreadline8:amd64 (8.0-4) ...
Setting up python3.8-minimal (3.8.10-0ubuntu1~20.04) ...
Setting up libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04) ...
Setting up python3.8 (3.8.10-0ubuntu1~20.04) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...

# python3.8 -c 'import sys; print(sys.path)'
['', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']

@PawelZ-RD
Copy link

As additional options, you could try python3.9 get-pip.py --isolated or python3.9 get-pip.py --target <the directory where you want pip installed>.

Flag --isolated results in pip installed in /usr/lib/python3.9/site-packages i.e. outside of sys.path

# python3.9 get-pip.py --isolated
Collecting pip
  Downloading pip-21.2.4-py3-none-any.whl (1.6 MB)
     |████████████████████████████████| 1.6 MB 21.6 MB/s
Collecting setuptools
  Downloading setuptools-57.4.0-py3-none-any.whl (819 kB)
     |████████████████████████████████| 819 kB 52.5 MB/s
Collecting wheel
  Downloading wheel-0.37.0-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel, setuptools, pip
Successfully installed pip-21.2.4 setuptools-57.4.0 wheel-0.37.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

# python3.9 -m pip --version
/usr/bin/python3.9: No module named pip

/usr/lib/python3.9/site-packages# python3.9 -m pip --version
pip 21.2.4 from /usr/lib/python3.9/site-packages/pip (python 3.9)

Flag --target works great!:

# python3.9 get-pip.py --target $(python3.9 -c 'import sys; print(sys.path[-1])')
Collecting pip
  Downloading pip-21.2.4-py3-none-any.whl (1.6 MB)
     |████████████████████████████████| 1.6 MB 20.2 MB/s
Collecting setuptools
  Downloading setuptools-57.4.0-py3-none-any.whl (819 kB)
     |████████████████████████████████| 819 kB 32.0 MB/s
Collecting wheel
  Downloading wheel-0.37.0-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel, setuptools, pip
Successfully installed pip-21.2.4 setuptools-57.4.0 wheel-0.37.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

# python3.9 -m pip --version
pip 21.2.4 from /usr/lib/python3/dist-packages/pip (python 3.9)

Could you update #usage of README.md document including description of all get-pip.py, please?

@pradyunsg
Copy link
Member

pradyunsg commented Aug 27, 2021

I believe this is a bug in Debian's Python package. Their modifications to Python have been a source of a long standing debate: https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e has a lot of discussion.

One of the pip maintainers has already filed an issue for Python 3.10 distutils/sysconfig -- https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1940705.

I strongly recommend filing an issue with Debian and Ubuntu for this -- while pip can do things to paper over the issue, the fundamental problem is that the Python installation is not proper. Part of the problem is that Debian users don't ask Debian's maintainers to make fixes for the things they break.

manually installing distutils

How did you do this? If you've used what is available in CPython's source tree, then you've installed an incompatible distutils for the Python interpreter -- Debian relies on patches they make to distutils to keep things working.

@PawelZ-RD
Copy link

PawelZ-RD commented Aug 27, 2021

I believe this is a bug in Debian's Python package. Their modifications to Python have been a source of a long standing debate: https://gist.github.com/tiran/2dec9e03c6f901814f6d1e8dad09528e has a lot of discussion.

I think that is the issue, Ubuntu packages are inconsistent, i.e. python3.8 and python3.9 bring different set of modules, as well as have different set of decencies, while all that should really differ is just Python version. Same applies for python3.[89]-minimal. However, all 4 are consentient in one thing - not having sys.prefix/lib/pythonX.Y/site-packages in sys.path
To have that (and distutils) sorted, one needs to install a collection of python3 packages, but those are Python3.8 (and bring tons of semi-random libraries and so).

All Ubuntu provided Python3.[89] 'installations' have ensurepip removed. I think, the 'logic' is to force people to use python3-pip that (unless --no-install-recommends is used) brings whole tone of things including make, cpp and perl(!)

I strongly recommend filing an issue with Debian and Ubuntu for this -- while pip can do things to paper over the issue, the fundamental problem is that the Python installation is not proper. Part of the problem is that Debian users don't ask Debian's maintainers to make fixes for the things they break.

I will consider that.

manually installing distutils

How did you do this? If you've used what is available in CPython's source tree, then you've installed an incompatible distutils for the Python interpreter -- Debian relies on patches they make to distutils to keep things working.

Installing might be a bit of overstatement, since I am working with Docker container I am doing:

COPY --from=python:3.9-slim /usr/local/lib/python3.9/distutils /usr/lib/python3.9/distutils

I would do FROM python:3.9-slim but I have to rely on specific 'base' container.
python:3.9-slim has FROM debian:bullseye-slim

@pradyunsg
Copy link
Member

Okay, so it's confirmed that this is caused by Debian's patches to Python.

They break out distutils from the CPython standard library. Please file a bug with Debian, with reportbug python3 (per their reporting documentation). You can link to this issue in your bug report.

Closing this, since I don't think we can do anything about this on our end. If someone has ideas, please go ahead and let us know in a new issue. :)

@ei-grad
Copy link

ei-grad commented Oct 14, 2021

Workaround worked for me:

ARG PYTHON=python3.10
RUN curl https://bootstrap.pypa.io/get-pip.py | \
        $PYTHON - pip==21.3 && \
        # pip adopts the behaviour which is unsupported by debian
        # https://github.com/pypa/get-pip/issues/124
        mkdir /usr/lib/$PYTHON/dist-packages && \
        echo /usr/lib/$PYTHON/site-packages > /usr/lib/$PYTHON/dist-packages/site-packages.pth && \
        rm -rf /tmp/*

@edwardnguyen1705
Copy link

Hi @PawelZ-RD ,
Could you tell me, how manually adding distutils?

@pradyunsg
Copy link
Member

/cc @doko42

@elkd
Copy link

elkd commented Apr 19, 2022

😭 Why Debian, Why Python? Why?

@ei-grad
Copy link

ei-grad commented Apr 19, 2022

https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1940705

Stefano Rivera (stefanor)] on 2022-03-12:
Fixed in 3.10.2-3.
Changed in python3.10 (Ubuntu):
status: Confirmed → Fix Released

@VioPsycho
Copy link

VioPsycho commented Apr 22, 2022

if you get this error
ModuleNotFoundError: No module named 'distutils.cmd'
you can try it
sudo apt-get install python3.10-distutils

@AhmedTremo
Copy link

This solved it for me:

sudo apt-get install --reinstall python3.7-distutils

@pradyunsg
Copy link
Member

Please report this as a bug to Debian and Ubuntu, if it has not been reported already.

@pypa pypa locked as resolved and limited conversation to collaborators Jun 12, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests