handle distutils deprecation

[dimbleby]

distutils is going away in python 3.12.

I don't find this comment very clear

# We can't use sysconfig.get_paths() because
# on some distributions it does not return the proper paths
# (those used by pip for instance).

but it was written three years ago now, we can always hope that things have moved on far enough to make it no longer a thing that needs to be worried about.

Well anyway let's start by seeing how the pipelines like it if we don't use distutils...

[dimbleby]

probably the comment is a reference to debian cf #6459 (comment)

on the other hand that open issue seems to say that things already don't work on such distributions unless you create a virtual environment - in which case this wouldn't make things worse anyway?

[radoering]

I did some basic testing on Ubuntu 20.04:

Test script

import site
import sysconfig
import warnings

from pprint import pprint

orig_paths = sysconfig.get_paths().copy()
if site.check_enableusersite():
  orig_paths["usersite"] = site.getusersitepackages()
  orig_paths["userbase"] = site.getuserbase()
print("sys_config.get_paths():")
pprint(orig_paths)
paths = orig_paths.copy()

from distutils.core import Distribution
from distutils.command.install import SCHEME_KEYS

d = Distribution()
d.parse_config_files()
with warnings.catch_warnings():
  warnings.filterwarnings("ignore", "setup.py install is deprecated")
  obj = d.get_command_obj("install", create=True)
obj.finalize_options()
for key in SCHEME_KEYS:
  if key == "headers":
      # headers is not a path returned by sysconfig.get_paths()
      continue

  paths[key] = getattr(obj, f"install_{key}")

if site.check_enableusersite() and hasattr(obj, "install_usersite"):
  paths["usersite"] = getattr(obj, "install_usersite")
  paths["userbase"] = getattr(obj, "install_userbase")

print("distutils:")
pprint(paths)

print(f"Equal? {paths == orig_paths}")

Python 3.8: ❌ (not equal)

sys_config.get_paths():
{'data': '/usr',ers for dbus (1.12.16-2ubuntu2.3) ...
'include': '/usr/include/python3.8',ls (0.136ubuntu6.7) ...
'platinclude': '/usr/include/python3.8',
'platlib': '/usr/lib/python3.8/site-packages',
'platstdlib': '/usr/lib/python3.8',
'purelib': '/usr/lib/python3.8/site-packages',
'scripts': '/usr/bin',
'stdlib': '/usr/lib/python3.8',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.8/site-packages'}
distutils:
{'data': '/usr/local',
'include': '/usr/include/python3.8',
'platinclude': '/usr/include/python3.8',
'platlib': '/usr/local/lib/python3.8/dist-packages',
'platstdlib': '/usr/lib/python3.8',
'purelib': '/usr/local/lib/python3.8/dist-packages',
'scripts': '/usr/local/bin',
'stdlib': '/usr/lib/python3.8',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.8/site-packages'}
Equal? False

Python 3.9: ❌ (not equal)

sys_config.get_paths():
{'data': '/usr',
'include': '/usr/include/python3.9',
'platinclude': '/usr/include/python3.9',
'platlib': '/usr/lib/python3.9/site-packages',
'platstdlib': '/usr/lib/python3.9',
'purelib': '/usr/lib/python3.9/site-packages',
'scripts': '/usr/bin',
'stdlib': '/usr/lib/python3.9',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.9/site-packages'}
distutils:
{'data': '/usr/local',
'include': '/usr/include/python3.9',
'platinclude': '/usr/include/python3.9',
'platlib': '/usr/local/lib/python3.9/dist-packages',
'platstdlib': '/usr/lib/python3.9',
'purelib': '/usr/local/lib/python3.9/dist-packages',
'scripts': '/usr/local/bin',
'stdlib': '/usr/lib/python3.9',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.9/site-packages'}
Equal? False

Python 3.10: ✔ (equal)

sys_config.get_paths():
{'data': '/usr/local',
'include': '/usr/include/python3.10',
'platinclude': '/usr/include/python3.10',
'platlib': '/usr/local/lib/python3.10/dist-packages',
'platstdlib': '/usr/lib/python3.10',
'purelib': '/usr/local/lib/python3.10/dist-packages',
'scripts': '/usr/local/bin',
'stdlib': '/usr/lib/python3.10',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.10/site-packages'}
/mnt/c/dev/poetry/sysconfig_paths.py:15: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils.core import Distribution
distutils:
{'data': '/usr/local',
'include': '/usr/include/python3.10',
'platinclude': '/usr/include/python3.10',
'platlib': '/usr/local/lib/python3.10/dist-packages',
'platstdlib': '/usr/lib/python3.10',
'purelib': '/usr/local/lib/python3.10/dist-packages',
'scripts': '/usr/local/bin',
'stdlib': '/usr/lib/python3.10',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.10/site-packages'}
Equal? True

Python 3.11: ✔ (equal)

sys_config.get_paths():
{'data': '/usr/local',
'include': '/usr/include/python3.11',
'platinclude': '/usr/include/python3.11',
'platlib': '/usr/local/lib/python3.11/dist-packages',
'platstdlib': '/usr/lib/python3.11',
'purelib': '/usr/local/lib/python3.11/dist-packages',
'scripts': '/usr/local/bin',
'stdlib': '/usr/lib/python3.11',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.11/site-packages'}
/mnt/c/dev/poetry/sysconfig_paths.py:15: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils.core import Distribution
distutils:
{'data': '/usr/local',
'include': '/usr/include/python3.11',
'platinclude': '/usr/include/python3.11',
'platlib': '/usr/local/lib/python3.11/dist-packages',
'platstdlib': '/usr/lib/python3.11',
'purelib': '/usr/local/lib/python3.11/dist-packages',
'scripts': '/usr/local/bin',
'stdlib': '/usr/lib/python3.11',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.11/site-packages'}
Equal? True

Python 3.9 venv: ✔ (equal)

sys_config.get_paths():
{'data': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9',
'include': '/usr/include/python3.9',
'platinclude': '/usr/include/python3.9',
'platlib': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9/lib/python3.9/site-packages',
'platstdlib': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9/lib/python3.9',
'purelib': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9/lib/python3.9/site-packages',
'scripts': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9/bin',
'stdlib': '/usr/lib/python3.9',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.9/site-packages'}
distutils:
{'data': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9',
'include': '/usr/include/python3.9',
'platinclude': '/usr/include/python3.9',
'platlib': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9/lib/python3.9/site-packages',
'platstdlib': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9/lib/python3.9',
'purelib': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9/lib/python3.9/site-packages',
'scripts': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9/bin',
'stdlib': '/usr/lib/python3.9',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.9/site-packages'}
Equal? True

on the other hand that open issue seems to say that things already don't work on such distributions unless you create a virtual environment - in which case this wouldn't make things worse anyway?

I agree and even if it worked I'd say it's a Debian/Ubuntu bug and the workaround is to create a virtual environment. I think I will check in the next maintainer meeting if there are other opinions and if we want to merge it now or later.

[dimbleby]

Indeed this has long been understood as a debian bug.

The muddiness, as I understand it, is that at some point the workarounds which I am removing in this MR did actually work. And then at some point they stopped working (which is where #6459 has got to).

So unconditionally removing them here is abandoning any pretence that poetry intends to repair those workarounds.

It would I suppose be possible, though awfully ugly, to (i) fix the workarounds and (ii) 'improve' them with "if python version less than 3.12" checks, or something like that. I don't intend to do any such thing myself, though of course if y'all think that's worthwhile I wouldn't - couldn't! - stand in your way.

[Secrus]

In my opinion, we should drop the special cases and just point people to 3.9-venv or 3.10+ solution.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.