Python is deprecating distutils.cfg

[carlocab]

If you have been directed here from a pip warning, you don't need to take any action at this stage. Nothing is currently expected to break until Python 3.12, and a full solution will be implemented well before then.

Note that as of 3.10.2, this issue no longer affects Python 3.10 and later.

---

Quoting @uranusjr from Homebrew/discussions#1405:

I am one of the maintainers of pip, the Python package installer. We are recently informed that Homebrew is currently using distutils.cfg to configure package installation schemes, and we believe it is best for Homebrew to remove the usage.

Python is currently deprecating distutils, and will remove the module entirely in 3.10 (PEP 632). As a part of the transition, pip is moving all its current distutils usages to the sysconfig module instead, and a user recently reported pip would no longer work correctly with Homebrew-compiled Python after the transision. This is due to Homebrew’s usage of distutils.cfg, which is no longer supported in sysconfig.

The distutils.cfg file (and its various variants) has long been disfavoured by Python core maintainers, and the recommendation, if redistributors wish to provide custom install schemes, has been to directly patch distutils and sysconfig instead. This is the current approach taken by all Linux distributions that we have contact with, and the community is working toward providing better support to the patching process (b.p.o-43976). I’m recommending Homebrew to also migrate away from the distutils.cfg usage, since it is going to break while the community continuously migrate away from distutils, and stop working entirely when Python 3.12 lands, and join the conversation with other Python redistributors to improve sysconfig to support Homebrew’s needs.

Reposting here so it won't get buried in the Discussions page. We probably want to fix python@3.9 so it can be easily re-used for python@3.10.

[Bo98]

has been to directly patch distutils and sysconfig instead

This links a bit with the pth situation, where it might also be benficial to patch to prevent pip uninstalling brew packages.

the community is working toward providing better support to the patching process (b.p.o-43976)

Perhaps best to wait for the outcome of this before acting.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[uranusjr]

As a related follow-up, a conversation happend during PyCon US last week between PyPA and Linux distribution folks around how the make distribution package managers (e.g. brew) and Python package managers (e.g. pip) work well together. We are working on an informational PEP to propose some new mechanisms and best practices, which may interest Homebrew maintainers. We plan to continue the conversaion on the linux-sig mailing list (not started yet) so it would be awesome if Homebrew maintainers could follow it as well (the list is called linux-sig for historical reasons, but the topic applies to many downstream Python distributors including Hombrew, MacPorts, Anaconda, etc.)

[carlocab]

Related: geofft/peps#3

[alebcay]

Hi @uranusjr, thanks for that info. I don't know about anyone else on the Homebrew team but I've joined the linux-sig list you linked for now. Reading through that draft PEP now (thanks Carlo).

[Bo98]

Just wait until a solution has been figured out?

For users: yes. AFAIK nothing is broken yet.

---

The warning still seems a bit early considering a proper solution forward has not yet been agreed in the Python community. The discussion in the linked draft PEP appears to have stalled for now.

It seems the pip team might be trying to push us into applying custom patches. This is not a long term solution at all and is not something we want to maintain in the long run.

Assuming sysconfig is what pip are wanting to use going forward, we actually already do "fix" some sys values via a sitecustomize.py file so I can look into seeing if it works for sysconfig perhaps, but I would like us to focus on getting a better solution integrated into Python 3.10 (or probably 3.11 at this point) as this is becoming a cat and mouse game with different tools using different things - it's not universally sysconfig sadly, even when distutils vanishes.

[SMillerDev]

It seems the pip team might be trying to push us into applying custom patches. This is not a long term solution at all and is not something we want to maintain in the long run.

IYAM it's not something we want to maintain in the short term either and if upstream prefers to ship a broken system, Homebrew's stance on patches makes it perfectly fine for us to ship said broken version.

[uranusjr]

I think the main problem is I failed to communicate the situation clearly, so please let me try to describe what we’re facing and hopefully provide enough context for the situation.

Python 3.12 is going to remove distutils, and along with it, support for distutils.cfg. This means that how Homebrew is currently distributing Python will break, and we need to do something to avoid Homebrew being unable to distribute a working Python 3.12 when it lands. The approach currently persumed by the Linux community could be a good way to resolve this, but they do not have the same hard deadline on 3.12 like Homebrew (although it is preferred if they could achieve this at that time), because they already have patches to Python that would work for both current Python versions and past 3.12. Homebrew therefore would need to either more actively drive that discussion forward, or persume a downstream patch solution like Linux distributions currently do to keep things working.

Pip, like Homebrew, is currently relying on distutils functionalities, and have the same 3.12 deadline. We are therefore trying to rewrite things in pip to keep it working on Python 3.12 onwards. However, some of the new code, since it can’t depend on distutils and distutils.cfg, is incompatible with how Homebrew distributes Python. This is problematic since Homebrew allows users to freely upgrade their pip version in a Hombrew-distributed Python, and once pip completely migrates off the current distutils implementation (sometime before 3.12 lands), those users upgrading to a new pip version will see pip break on their Hombrew-distributed Python, and pip is going to face an impossible problem—we won’t be able to support a Homebrew-distributed Python because it uses mechanisms pip can’t use, but also can’t stop people from upgrading pip agaist those Python installations.

This is what the warning is about: to raise awareness to the issue, and hopefully gather enough resources and contribute to a resolution. We want pip to work on both Python 3.12 (Homebrew or otherwise), and a Python (<3.12) distributed by Homebrew, but this is not possible without help from Homebrew’s Python distribution maintainers. (Note that pip developers are not CPython developers, so we are not in the best position to come up with a solution that suits Homebrew best, although we are definitely good at checking whether a proposed solution would work!)

[SMillerDev]

but this is not possible without help from Homebrew’s Python distribution maintainers.

That's gonna be really hard because Homebrew doesn't have a concept of package maintainers. All Homebrew maintainers maintain all packages (or the brew tool) and the public can modify packages however they see fit, as long as it's within the rules defined in docs.brew.sh

[uranusjr]

Then I guess that sort of enhance my belief we should keep the warning until we can find someone in the public willing to come up with something :)

Please do also provide any suggestions how we could improve the warning’s presentation to maximise the effect as well!

[rgoldberg]

Would the Allow Python distributors to add custom site install schemes feature make it easy enough for Homebrew (& others) to switch from distutils to sysconfig?

If not, what can the Python / sysconfig people do to make Homebrew's (& others') transitions easier?

If so, wouldn't it make sense for the Python / sysconfig maintainers to implement that feature before removing distutils?

[uranusjr]

What should we do know?

Nothing for now since pip is not going to break anything for existing Python versions. But you probably want to come back to this issue before Homebrew tries to upgrade Python to 3.11 (or more critically 3.12) and see if the issue is resolved yet and decide if you want to apply the upgrade.

If so, wouldn't it make sense for the Python / sysconfig maintainers to implement that feature before removing distutils?

Of course it does, but in the end it’s up to the downstream distributors to tell CPython what they want. If no-one says anything, CPython maintainers can only assume everyone is fine with the removal. (Which is what the b.p.o. issue you linked is expressing; FFY00 is the Python distributor for Arch Linux IIRC, so CPython maintainers are trying to understand their needs and adjust sysconfig accordingly.)

[rgoldberg]

it’s up to the downstream distributors to tell CPython what they want

Has anyone from Homebrew told CPython what they want / need?

If not, why?

If so, will CPython provide it?

If they won't provide it, why?

If they will provide it, what is left to discuss? (I assume this isn't the scenario, since discussion would presumably have ended if it were so)

[Bo98]

Has anyone from Homebrew told CPython what they want / need?

Currently we provide a distutils.cfg as such:

[install]
prefix=/usr/local
[build_ext]
include_dirs=/usr/local/include:/usr/local/opt/openssl@1.1/include:/usr/local/opt/sqlite/include
library_dirs=/usr/local/lib:/usr/local/opt/openssl@1.1/lib:/usr/local/opt/sqlite/lib

To my understanding, the way to configure build_ext hasn't changed yet(?), though I've not been closely following setuptools changes. For pip, the discussion is mostly about the install config items, which can now be done by configuring sysconfig (even for things still using distutils once Python 3.10 releases).

Has anyone from Homebrew told CPython what they want / need?

It's no different than bpo-43976, so there's not much to add there. Having a site-packages that lies in a different prefix to the stdlib prefix isn't a unique situation.

I'm happy however to be involved in any necessary discussions however.

what is left to discuss?

Honestly, I don't know. I can leave a comment as a "poke" but maybe that's more annoying than helpful.

[rgoldberg]

It's no different than bpo-43976, so there's not much to add there. Having a site-packages that lies in a different prefix to the stdlib prefix isn't a unique situation.

bpo-43976 seems to indicate that Homebrew will have to implement a patch for the Python install. You (@Bo98) & @SMillerDev indicated that you don't want to do that for the long term, and possibly not even for the short term. Is that correct?

@uranusjr If a patch would be required, would it be just some simple boilerplate around a few settings, rather than a significant effort? If so, would the boilerplate be stable for a long time, or might it need to be frequently changed for new Python versions?

If the patch would be fairly simple & stable, would the Homebrew maintainers be amenable to using such a patch? If you aren't amenable to using a patch that is both simple & stable, why?

If the patch should be fairly simple, could someone from CPython / Python / pip / whatever provide a simple patch with placeholders for the values that Homebrew (or others) would supply? Could they commit to providing updates for the simple patch of it must be modified for a new Python version?

If a simple patch would cover most downstream use cases, could CPython (or whoever) just provide the equivalent of that patch either as install options for the values themselves and/or as an install option to specify a config file that provides the values?

Sorry if this has all been answered before, but there are too many comments to quickly find the current status.

Can the current status of everything be inserted into the initial issue comment, so everyone can get up to speed quickly?

Thanks for any info.

[Bo98]

bpo-43976 seems to indicate that Homebrew will have to implement a patch for the Python install.

No, bpo-43976 is a request to add a system to customise without applying patches. We don't consider adding new files as a patch (we already do this with sitecustomize.py and distutils.cfg).

[Bo98]

Can the current status of everything be inserted into the initial issue comment, so everyone can get up to speed quickly?

I've added a short note for end users seeing the pip warning, since that'll be what most of the traffic here will be. People interested in the implementation details can follow along with the comments.

[uranusjr]

@Bo98 I am not faimilar with the situation for setuptools either, but setuptools is known for keeping compatibility above almost everything, so I wouldn’t be surprised if they keep supporting distutils.cfg until the end of time. But you are likely still going to need to change something, because distutils.cfg should currently be placed in {python_prefix}/lib/{python_version}/distutils, which is going to disappear entirely in 3.12, so at the very least that file needs to go somewhere else.

@rgoldberg See Debian’s patch. Essentially you need make get_default_scheme return something else than the default scheme (which would be under the default prefix e.g. /usr/local/opt/python@3.10). I’m not sure if Homebrew needs to be able to install Python packages to that default prefix; if it doesn’t, maybe Homebrew can simply patch out posix_prefix to point to /usr/local (but remember to account for virtual environments; which can be detected by checking sys.prefix and sys.base_prefix). If the ability to install into the default prefix must be kept like Debian (their Python’s default prefix is /usr, which is also the prefix for e.g. apt install python3-pip, but they want pip to use /usr/local as prefix), then you need one additional scheme and some mechanism to switch between them (Debian uses an environment variable).

I think this also somewhat answers your other question about why there is not a commonly maintained patch, or even upstreamed back into CPython. Each downstream distributor does things a bit differently, so their patches also need to slightly differ, and they can’t agree on what to upstream in the first place (the metioned in-draft PEP is an effort toward some kind of common ground, but it’s just very hard). I believe Red Hat’s patch is actually more popular than Debian’s (also used by at least Cygwin and MSYS2), but their setups are also significantly different from Homebrew’s. Homebrew’s setup is (from my very limited experience) actually most similar to NixOS(?) but Nix does some very fancy stuff because they have very ambitious goals not perused by anyone else.

[rgoldberg]

bpo-43976 seems to indicate that Homebrew will have to implement a patch for the Python install.

No, bpo-43976 is a request to add a system to customise without applying patches. We don't consider adding new files as a patch (we already do this with sitecustomize.py and distutils.cfg).

Sorry. I only scanned the issues & its comments. I saw some talk in the comments about "patching" sysconfig.py, so I assumed that people were suggesting patching it instead of providing a proper config file.

[rgoldberg]

See Debian’s patch

@uranusjr must Debian's patch be updated frequently for new Python versions, or is it fairly stable?

If Python requires patching instead of supporting a config file, if a patch similar to Debian's will work for Homebrew & would be stable for many Python versions, 3 dozen or so line changes in a single file to get Python working in Homebrew seems preferable to providing it broken, and to not providing new versions at all. It might not be as ideal as Python supporting a config file, but it's better than the alternatives, if Python forces patching upon everyone.

[uranusjr]

From what I hear, the patch is pretty stable.

[iaxat]

Does anyone know when this issue will be solved?

[nevertoday]

how to fixed this ?

[marcuss]

how to fixed this ?

I had to literally go back to python3.9 and now I see this error, apparently the project I'm working on could not be upgraded, how do we fix this?

[Clivern]

I am getting lots of errors on homebrew because of this. Is there a way to hide these warnings on mac meanwhile?

[houchaowei]

The solution to this problem I understand is to upgrade 3.9->3.10+ on it?

[davvid]

Related: #152017

[iMichka]

@Bo98 what do you think about closing this issue? I feel that we will never « fix » this in Python@3.9 (and to be fair, that’s totally OK).

Most of our formulae have migrated to newer Python versions and Python@3.9 downloads are slowly decreasing. The EOL date is 2025-10, so users will have to survive with current state until then.

[MikeMcQuaid]

Yeh, let's close, thanks @carlocab @iMichka and everyone else who helped here!