Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre-installed Python conflicts with Homebrew Python on macOS #9966

Open
4 of 14 tasks
bebound opened this issue May 31, 2024 · 6 comments
Open
4 of 14 tasks

Pre-installed Python conflicts with Homebrew Python on macOS #9966

bebound opened this issue May 31, 2024 · 6 comments

Comments

@bebound
Copy link

bebound commented May 31, 2024

Description

In macOS 12 and 13, it has pre-installed Python /Library/Frameworks/Python.framework/Versions/3.xx/, which creates soft link such as /usr/local/bin/2to3-3.xx. Homebrew's python@3.xx also wants to create this link.

This causes issue when there is new Python is released, and brew install xx tries to install the latest Python: see Homebrew/homebrew-core#165793 (comment)

Runner image has a fix for Python 2.7:

choice_changes_xml=$(mktemp /tmp/python2_choice_changes.xml.XXXXXX)
sudo installer -showChoiceChangesXML -pkg $python2_pkg -target / | tee $choice_changes_xml > /dev/null
# To avoid symlink conflicts, remove tools installation in /usr/local/bin using installer choices
xmllint --shell $choice_changes_xml <<EOF
cd //array/dict[string[text()='org.python.Python.PythonUnixTools-2.7']]/integer
set 0
save
EOF

It used to let brew overwrite the python@3.11 soft link to resolve the conflict, but is removed in https://github.com/actions/runner-images/pull/8452/files#diff-5490845eec9cd46fa52e10c62bd6ec6a76c27f17a338963a508414efd8b2252dR33-R34

Now I have to run brew unlink python@3.11 && brew link --overwrite python@3.11 before using brew install xxx to eliminate the potential conflict. It's tedious.

There is a PR attempting to address this issue: #7710, but it was rejected. Regardless of whether the fix should be implemented in actions/setup-python, we are still facing this issue one year later. I believe the runner image should consider providing a temporary fix for this.

Related issue:
Homebrew/homebrew-core#173191
Azure/azure-cli#29054

Platforms affected

  • Azure DevOps
  • GitHub Actions - Standard Runners
  • GitHub Actions - Larger Runners

Runner images affected

  • Ubuntu 20.04
  • Ubuntu 22.04
  • Ubuntu 24.04
  • macOS 11
  • macOS 12
  • macOS 13
  • macOS 13 Arm64
  • macOS 14
  • macOS 14 Arm64
  • Windows Server 2019
  • Windows Server 2022

Image version and build link

This could be reproduced by reinstalling python@3.11

brew uninstall azure-cli
brew install python@3.11

https://github.com/bebound/starter-workflows/actions/runs/9298969591/job/25591905178

Is it regression?

Yes?

Expected behavior

Successfully install python@3.11 with brew.

Actual behavior

Pouring python@3.11--3.11.9.ventura.bottle.tar.gz
Error: The `brew link` step did not complete successfully
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/2to3-3.[11](https://github.com/bebound/starter-workflows/actions/runs/9298969591/job/25591905178#step:7:12)
Target /usr/local/bin/2to3-3.11
already exists. You may want to remove it:
  rm '/usr/local/bin/2to3-3.11'

To force the link and overwrite all conflicting files:
  brew link --overwrite python@3.11

To list all files that would be deleted:
  brew link --overwrite python@3.11 --dry-run

Possible conflicting files are:
/usr/local/bin/2to3-3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3-3.11
/usr/local/bin/idle3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/idle3.11
/usr/local/bin/pydoc3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/pydoc3.11
/usr/local/bin/python3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11
/usr/local/bin/python3.11-config -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11-config

Repro steps

brew uninstall azure-cli
brew install python@3.11

@Alexey-Ayupov
Copy link
Collaborator

Hello @bebound, this issue should be fixed after the next image release.

@bebound
Copy link
Author

bebound commented Jun 4, 2024

@Alexey-Ayupov May I know which PR fixes this?

@erik-bershel
Copy link
Contributor

Hello @bebound !

A little (not exactly true 😄) clarification. @Alexey-Ayupov meant that the problem would resolve itself with updating the macOS images until Homebrew released dependency updates.
Unfortunately this problem lies between setup-python and Homebrew - their packages conflict due to the paths used as @cho-m mentioned.
Hypothetically, we could change the path, which would remove the problem for Homebrew users, but would break users of the setup-python action for some scenarios. Unfortunately, the most complete solution to this problem “forever” would be to remove the Python packages from the image in order to install it during runtime from specific source by customers, which is a bit irrational. Relinking is always easier than reinstalling.

We will do some more research on what can be done in a way not to break existed workflows and welcome any ideas from you as a user.

@bebound
Copy link
Author

bebound commented Jun 6, 2024

I prefer let homebrew overwrite setup-python for two reasons:

  1. Homebrew is more likely to trigger the conflict as it always checks for new version, and it's hard to debug when brew install fails suddenly because Python release a new version.
  2. It's more explicit/efficient to resolve the conflict when users use setup-python than use Brew to install packages depending on python@xxx.

Also, if this possible to pre-install brew's python@xx to replace setup-python on macOS?

Out of curiosity, which line creates the soft link from /usr/local/bin/python3.11 to /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11?

@QuLogic
Copy link

QuLogic commented Jun 17, 2024

It seems like this is slowly rolling out to more runners.

@QuLogic
Copy link

QuLogic commented Aug 14, 2024

Oh, and it's back again now that Homebrew released Python 3.12.5, but the images are on an older version. It's also affecting both Azure and GitHub Actions now.

The annoying thing is that we aren't even using the Homebrew Python, but the one from setup-python/UsePythonVersion. But one of our system dependencies (installed via Homebrew) does, and so we have this conflict arise for something we don't really use.

MackBambu added a commit to MackBambu/BambuStudio that referenced this issue Aug 15, 2024
1.openvdb debug mode build fail;
2.github action macos build error, rm '/usr/local/bin/2to3-3.11' actions/runner-images#9966
3.Modify the wrong trigger CI branch option
jira:nojira
radioactiveman added a commit to audacious-media-player/audacious-plugins that referenced this issue Aug 19, 2024
If Python is installed/updated without "--overwrite",
conflict errors occur because Homebrew tries to replace
existing files like "/usr/local/bin/2to3".

See also:
- actions/runner-images#9966
@erik-bershel erik-bershel removed the awaiting-deployment Code complete; awaiting deployment and/or deployment in progress label Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants