brew update followed by brew upgrade fails for Python's 2to3 conflict (December 2022)

[traversaro]

Description

If one has on a job:

brew update
brew upgrade

the job fails with error:

 ==> Upgrading python@3.10
  3.10.8 -> 3.10.9 

==> Pouring python@3.10--3.10.9.big_sur.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/2to3
Target /usr/local/bin/2to3
already exists. You may want to remove it:
  rm '/usr/local/bin/2to3'

To force the link and overwrite all conflicting files:
  brew link --overwrite python@3.10

To list all files that would be deleted:
  brew link --overwrite --dry-run python@3.10

Possible conflicting files are:
/usr/local/bin/2to3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3
/usr/local/bin/idle3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/idle3
/usr/local/bin/pydoc3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/pydoc3
/usr/local/bin/python3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3
/usr/local/bin/python3-config -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3-config

This is similar to old issues:

• brew upgrade fails for Python's 2to3 conflict  runner-images#2322
• Issue with brew runner-images#4020

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 18.04
• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• Windows Server 2019
• Windows Server 2022

Image version and build link

Image Version: 20221215.1
Link: https://github.com/traversaro/github-actions-brew-update-upgrade-check/actions/runs/3742142787

Is it regression?

20221211.1

Expected behavior

The brew upgrade should exit fine.

Actual behavior

The brew upgrade fails with the error message in the top of the issue.

Repro steps

See https://github.com/traversaro/github-actions-brew-update-upgrade-check/blob/2fa847c627c643716ba10d681ea18c86c0b54dec/.github/workflows/brew-update.yml for a MWE .

[traversaro]

Possibly related to actions/runner-images#6459 .

[igorboskovic3]

Hi @traversaro , thank you, we will take a look.

[david-engelmann]

I'm observing this as well

[ryancurrah]

Also observed this here https://github.com/ryancurrah/lima-and-qemu/actions/runs/3745432855/jobs/6359863993 and here https://github.com/mook-as/lima-and-qemu/actions/runs/3743467832/jobs/6356414606.

EDIT:

I was able to get it working by removing the existing symlinks.

rm /usr/local/bin/2to3
rm /usr/local/bin/idle3
rm /usr/local/bin/pydoc3
rm /usr/local/bin/python3
rm /usr/local/bin/python3-config

[mikhailkoliada]

It looks like it the universal python's package used in actions/setup-python is installing its symlinks to /usr/local, while I think it should not.

@marko-zivic-93 @dsame @dmitry-shibanov could you please take a look?

[ni4]

Got the same issue, and find out that something similar we had previously. Now it looks like:

  # homebrew fails to update python 3.9.1 to 3.9.1.1 due to unlinking failure
  rm /usr/local/bin/2to3 || true
  # homebrew fails to update python from 3.9 to 3.10 due to another unlinking failure
  rm /usr/local/bin/idle3 || true
  rm /usr/local/bin/pydoc3 || true
  rm /usr/local/bin/python3 || true
  rm /usr/local/bin/python3-config || true

[Flamefire]

It looks like it the universal python's package used in actions/setup-python is installing its symlinks to /usr/local, while I think it should not.

That is not the case for us and the OP:

/usr/local/bin/2to3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3

There is seemingly a non-brew installed Python from somewhere else

[dsame]

@Flamefire the workflow does not use action/setup-python. It seems the links are created by the macos python package - i see them on the "almost" clean machine with the XCode installed.

[dsame]

@traversaro actully it is a normal behaviour. As you see from the log the installation continues normally with just a warning the links were not overwritten. The problem is brew upgrade exits with non-zero exit code and bash shell runs with set -e option. Thus the step signals the error condition despite brew upgrade is the very last command or not.

The easiest way to make the step to success is to add true after brew upgrade:

      - run: |
          brew update
          brew upgrade || true

[dsame]

Hello @fredroy, i see you forces the links by removing them manually in fact it is better to be done with

brew link --overwrite python@3.10

[maxkratz]

I'm also observing this behavior (just with running brew update and macOS 12): https://github.com/eMoflon/emoflon-ibex-eclipse-build/actions/runs/3764385410/jobs/6398766536

[Flamefire]

@dsame

@Flamefire the workflow does not use action/setup-python. It seems the links are created by the macos python package - i see them on the "almost" clean machine with the XCode installed.

Yes this is what I also observed and wrote in #577

As you see from the log the installation continues normally with just a warning the links were not overwritten. The problem is brew upgrade exits with non-zero exit code

We also see this when we install a package with brew (e.g. ccache) which requires Python and hence is automatically upgraded. Ignoring the exit code is not feasible in that case as it might hint to anything besides this issue.

i see you forces the links by removing them manually in fact it is better to be done with

brew link --overwrite python@3.10

This is also not feasible for us as the exact Python version required for a package to be installed is not known/may change.

Hence our workaround is to remove the symlinks on failure and retry: https://github.com/boostorg/boost-ci/blob/ac5ae7e901e49351d19de278cbf82ce8d2c44216/ci/setup_ccache.sh#L17-L24

@maxkratz

I'm also observing this behavior (just with running brew update and macOS 12): https://github.com/eMoflon/emoflon-ibex-eclipse-build/actions/runs/3764385410/jobs/6398766536

No, I also made this false assumption. The behavior is not triggered by the update (which just downloads meta information similar to apt-get update) but the following installation of a package: brew install p7zip coreutils grep wget curl (or brew upgrade but you don't use that here)

[maxkratz]

@maxkratz

I'm also observing this behavior (just with running brew update and macOS 12): https://github.com/eMoflon/emoflon-ibex-eclipse-build/actions/runs/3764385410/jobs/6398766536

No, I also made this false assumption. The behavior is not triggered by the update (which just downloads meta information similar to apt-get update) but the following installation of a package: brew install p7zip coreutils grep wget curl (or brew upgrade but you don't use that here)

Thank you for the clarification, you are right.

[aparnajyothi-y]

Hello Everyone, This behavior is expected. The installation proceeds normally with just a warning that the links were not overwritten, which is typical. The issue arises because the brew upgrade command exits with a non-zero exit code, and since the bash shell runs with the set -e option, it causes the step to fail even though brew upgrade is the final command.

Regarding the versions, Python 2.7, 3.10.8, and 3.11.0 symlinks are created during image generation. Specifically:

Python 2.7 is linked to macOS Library/Frameworks
Python 3.10 is linked to the Homebrew cellar
Python 3.11 is linked to macOS Library/Frameworks
The possible resolution of this issue would be updating the image with latest patches of Python 3.10 and Python 3.11. instead of 3.10.8 and 3.11.0. All macOS-12, 13, and 14 images have been updated with the latest patches for these versions, which resolved the issue.

Please let us know if any further assistance or clarification!

[eli-schwartz]

@aparnajyothi-y,

You seem to have fundamentally failed to understand the entire reason anyone is unhappy with GitHub Actions.

The behavior is detrimental and bad. We know that GitHub expects this to happen and we don't wish for further clarification.

We wish for GitHub to repent, to acknowledge that the expected behavior is detrimental and bad, and for GitHub to redesign the Actions image such that the Actions image works robustly, which it currently does not.

It is an antipattern to respond to an issue saying that some behavior is bad and causes troubles, by responding "we know it's bad and causes troubles, but that was intentional".

[poshul]

The following works for us:

   - name: Force update all brew python
      run: |
        brew update
        # GHA messes with links https://github.com/actions/setup-python/issues/577 so we just force replace their installs' with brew's
        for pkg in $(brew list --formula | grep -E '^python(@[0-9]+\.[0-9]+)?$'); do
            brew upgrade --force --overwrite "$pkg"
        done
        brew cleanup