Python CLI Tools Installing Inside Python Framework on macOS-12 #6507

aetherealtech · 2022-11-02T17:15:31Z

Description

We have a GitHub Action that installs codemagic-cli-tools using the following shell command:

pip3 install codemagic-cli-tools

This is supposed to add several CLI programs to the path (installing them to usr/local/bin). We use one of them, app-store-connect, later in the action.

This script broke somewhere between 6 and 1 days ago (the last successful run was on 10/27, around 23:30 UTC, first failed run was yesterday), with the failure:

app-store-connect: command not found

After running some tests, we determined that none of the CLI tools that are supposed to be added are present in usr/local/bin. There was an update to codemagic-cli-tools on the first failed run, so we tried using the exact same version, and the error persisted.

After some more investigation, we noticed that on the last successful run, the python CLI was referenced as simply python3.10, but on the failing runs, it is referenced by the full path:

/Library/Frameworks/Python.framework/Versions/3.11/bin/python

So then we looked into the following directory:

/Library/Frameworks/Python.framework/Versions/3.11/bin

And we found the CLI tools installed there.

We tried running them from there, and got a bunch of Python errors, but that may be a problem with those tools.

However, it seems like the latest macos runner that upgraded Python from 3.10.8 to 3.11 resulted in something about the paths getting messed up for Python.

Last successful run: https://github.com/movehq/hq-driver-ios/actions/runs/3341642377/jobs/5533052139
First failing run: https://github.com/movehq/hq-driver-ios/actions/runs/3370254410/jobs/5590925123

Platforms affected

Azure DevOps
GitHub Actions - Standard Runners
GitHub Actions - Larger Runners

Runner images affected

Image version and build link

https://github.com/actions/runner-images/blob/macOS-12/20221027.1/images/macos/macos-12-Readme.md

Is it regression?

https://github.com/actions/runner-images/blob/macOS-12/20221018.2/images/macos/macos-12-Readme.md

Expected behavior

CLI tools installed by this Python library should be in usr/local/bin, so they are in the shell path.

Actual behavior

CLI tools are installed to /Library/Frameworks/Python.framework/Versions/3.11/bin, and not in the shell path.

Repro steps

Run the following script in a GitHub Action:

pip3 install codemagic-cli-tools
which app-store-connect

The text was updated successfully, but these errors were encountered:

See: actions/runner-images#6507

ssrobins · 2022-11-03T04:49:09Z

I'm seeing the same thing with pip3 install conan. I can work around it by setting the following step prior to any steps that use it:

- name: Add Python bin to PATH
  run: echo "/Library/Frameworks/Python.framework/Versions/Current/bin" >> $GITHUB_PATH

One of my CI jobs is scheduled to run every day. Last time it worked, the runner image was 20221018.2. First failure had runner image 20221027.1.

ssrobins · 2022-11-03T05:12:23Z

I bet this will fix the image:
c7958d7

I held off on a PR since the Contribution guide said macOS external contributions aren't supported, but let me know if you want me to make one.

igorboskovic3 · 2022-11-03T08:20:28Z

Hi @aetherealtech and @ssrobins thank you for reporting and contributing, we will check it and get back to you.

* lestarch: bumping zmq and GDS versions * lestarch: adding `wheel` to setup * lestarch: checking tools before leaving action * lestarch: reverting to mac11 runners See: actions/runner-images#6507

andreujuanc · 2022-11-03T20:52:06Z

Please give top priority to this, we need to issue a patch on iOS ASAP

al-cheb · 2022-11-04T08:08:03Z

Please give top priority to this, we need to issue a patch on iOS ASAP

Our recommendation is to use setup-python action - https://github.com/actions/setup-python

andreujuanc · 2022-11-04T10:35:14Z

Thanks @al-cheb team sends you a hug!
To the github team: I opened a ticket because we spent some minutes figuring this out and we would like us and everyone else on the platform to get reimbursed for this. Specially because MacOS minutes have a 10x multiplier.

… on macOS This way each time a new Python version is installed via Homebrew , we don't get CI failures due to the upstream Python distribution also being installed. Ref: actions/runner-images#6459 Ref: actions/runner-images#6507 Ref: actions/runner-images#2322

We currently use any system-installed Python version to invoke the wheel building process (which then happens inside a container). The exact Python version does not matter, but it must be a working Python version -- and that's currently not the case for the macos11 builds: Python and the Python installation directory are not inside PATH, hence we install nox, but it's not found in the next step. actions/runner-images#6518 fixes the problem, but is not yet rolled out universally. It seems like using setup-python even for this case is recommended (actions/runner-images#6507 (comment)), so let's do that.

Because of actions/runner-images#6507 (comment) Installing and running Python packages is no longer straightforward Even with a temporary fix, rclpy was no longer building on MacOS

mikhailkoliada · 2022-11-25T08:59:43Z

Deployed

Related: * macos-latest can currently be associated with either macos-11 or macos-12, so pinned it for the moment --> https://github.blog/changelog/2022-10-03-github-actions-jobs-running-on-macos-latest-are-now-running-on-macos-12/ * Update checkout actions and setup python actions versions * Tools not installed in PATH locations: actions/runner-images#6507 * Explicit export workaround for the only case that seems to fail (for PATH not found reasons): add PATH missing to PATH if os is macos-12 (the one pinned) and running Python 3.11

The github mac runner images added python 3.11 but, unlike 3.10, not via brew. This causes brew install to fail with link conflict errors now that boost-python3 switched over to 3.11. Workaround this, and future python brew link errors, by ovewriting all links for python packages. Homebrew/homebrew-core@8c79089 https://github.com/orgs/Homebrew/discussions/3895 actions/setup-python#577 actions/runner-images#6459 actions/runner-images#6507 actions/runner-images#2322

The github mac runner images added python 3.11 but, unlike 3.10, not via brew. This causes brew install to fail with link conflict errors now that boost-python3 switched over to 3.11. Workaround this, and future python brew link errors, by overwriting all links for python packages. Homebrew/homebrew-core@8c79089 https://github.com/orgs/Homebrew/discussions/3895 actions/setup-python#577 actions/runner-images#6459 actions/runner-images#6507 actions/runner-images#2322

… on macOS This way each time a new Python version is installed via Homebrew , we don't get CI failures due to the upstream Python distribution also being installed. Ref: actions/runner-images#6459 Ref: actions/runner-images#6507 Ref: actions/runner-images#2322

aetherealtech added bug report needs triage labels Nov 2, 2022

aetherealtech mentioned this issue Nov 2, 2022

Crash in app-store-connect on Python 3.11 codemagic-ci-cd/cli-tools#282

Closed

LeStarch added a commit to LeStarch/fprime that referenced this issue Nov 3, 2022

lestarch: reverting to mac11 runners

96da42c

See: actions/runner-images#6507

LeStarch mentioned this issue Nov 3, 2022

lestarch: bumping zmq and GDS versions nasa/fprime#1756

Merged

igorboskovic3 added OS: macOS investigate Collect additional information, like space on disk, other tool incompatibilities etc. and removed needs triage labels Nov 3, 2022

t3chguy mentioned this issue Nov 3, 2022

Add eslint rule unicorn/no-instanceof-array element-hq/element-desktop#434

Merged

jaisnan mentioned this issue Nov 3, 2022

Fix cbmc-viewer ci issues with Mac11 model-checking/kani#1844

Merged

4 tasks

igorboskovic3 mentioned this issue Nov 4, 2022

Add Python to $PATH #6518

Merged

4 tasks

igorboskovic3 self-assigned this Nov 4, 2022

Yesterday17 mentioned this issue Nov 6, 2022

Road to Annix 1.0 Release ProjectAnni/annix#7

Open

StarWitch mentioned this issue Nov 6, 2022

Release Automation taisei-project/taisei#289

Merged

priitlatt mentioned this issue Nov 7, 2022

keychain: command not found codemagic-ci-cd/cli-tools#285

Closed

al-cheb added the awaiting-deployment Code complete; awaiting deployment and/or deployment in progress label Nov 9, 2022

FranzPoize added a commit to ShredEagle/reusable-workflows that referenced this issue Nov 9, 2022

should fix python path issues actions/runner-images#6507

20aebaf

FranzPoize added a commit to ShredEagle/reusable-workflows that referenced this issue Nov 9, 2022

should fix python path issues actions/runner-images#6507

846f338

FranzPoize added a commit to ShredEagle/reusable-workflows that referenced this issue Nov 9, 2022

should fix python path issues actions/runner-images#6507

0266693

eli-schwartz mentioned this issue Nov 9, 2022

CI: fix macOS runners being broken mesonbuild/wrapdb#745

Merged

jeeb mentioned this issue Nov 11, 2022

github/workflows: force deletion of existing upstream python symlinks on macOS mpv-player/mpv#10849

Merged

kleisauke mentioned this issue Nov 13, 2022

pip3 installed software is not available in PATH in macos-12 #6496

Closed

11 tasks

imphil mentioned this issue Nov 13, 2022

CI: Explicitly setup Python to build wheels cocotb/cocotb#3125

Merged

kleisauke mentioned this issue Nov 13, 2022

Revert "macOS: install Meson and Ninja via Homebrew (#163)" lovell/sharp-libvips#167

Merged

Guillaumebeuzeboc mentioned this issue Nov 16, 2022

fix(CI): remove MacOS CI ubuntu-robotics/nodl#48

Merged

mikhailkoliada closed this as completed Nov 25, 2022

q3aiml mentioned this issue Jan 8, 2023

fix brew link error on mac CI ledger/ledger#2161

Merged

Adnn pushed a commit to ShredEagle/reusable-workflows that referenced this issue Jan 18, 2023

should fix python path issues actions/runner-images#6507

4ab53a3

MrAlex94 mentioned this issue Oct 9, 2023

watchman 2023.10.09.00 fails to install on macOS 13 with brew #8500

Closed

10 tasks

wusyong mentioned this issue Dec 14, 2023

fix: macOS CI build errors servo/mozjs#440

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python CLI Tools Installing Inside Python Framework on macOS-12 #6507

Python CLI Tools Installing Inside Python Framework on macOS-12 #6507

aetherealtech commented Nov 2, 2022

ssrobins commented Nov 3, 2022

ssrobins commented Nov 3, 2022 •

edited

Loading

igorboskovic3 commented Nov 3, 2022 •

edited

Loading

andreujuanc commented Nov 3, 2022

al-cheb commented Nov 4, 2022

andreujuanc commented Nov 4, 2022

mikhailkoliada commented Nov 25, 2022

Python CLI Tools Installing Inside Python Framework on macOS-12 #6507

Python CLI Tools Installing Inside Python Framework on macOS-12 #6507

Comments

aetherealtech commented Nov 2, 2022

Description

Platforms affected

Runner images affected

Image version and build link

Is it regression?

Expected behavior

Actual behavior

Repro steps

ssrobins commented Nov 3, 2022

ssrobins commented Nov 3, 2022 • edited Loading

igorboskovic3 commented Nov 3, 2022 • edited Loading

andreujuanc commented Nov 3, 2022

al-cheb commented Nov 4, 2022

andreujuanc commented Nov 4, 2022

mikhailkoliada commented Nov 25, 2022

ssrobins commented Nov 3, 2022 •

edited

Loading

igorboskovic3 commented Nov 3, 2022 •

edited

Loading