In a fresh miniconda environment, pip detects the macOS version number as 10.16, causing the new 1.17.0 onnxruntime wheels to not be found

[joshuacwnewton]

Describe the issue

We have a GitHub Actions workflow that performs the following steps:

• Installs a set of Python packages on an Ubuntu runner.
• Freezes all installed packages using pip freeze.
• Installs the frozen packages across multiple OSs (macOS, Windows, etc.)

The macos-11 runner began failing after the January 31st release of 1.17.0 with the following error:

ERROR: Could not find a version that satisfies the requirement onnxruntime==1.17.0 (from versions: 1.7.0, 1.8.0, 1.8.1, 1.9.0, 1.10.0, 1.11.0, 1.11.1, 1.12.0, 1.12.1, 1.13.1, 1.14.1, 1.15.0, 1.15.1, 1.16.0, 1.16.1, 1.16.2, 1.16.3)
ERROR: No matching distribution found for onnxruntime==1.17.0

The Ubuntu/Windows runners seem to be able to install the frozen requirement onnxruntime==1.17.0 just fine (alongside all of the other requirements we freeze). So, I don't think that this is a dependency conflict with another package.

If I had to guess, this might be because of the change in platform signifiers for macOS wheels:

• 1.16.3: onnxruntime-1.16.3-cp38-cp38-macosx_10_15_x86_64.whl
• 1.17.0: onnxruntime-1.17.0-cp38-cp38-macosx_11_0_universal2.whl

But, the docs on universal2 seem to suggest we meet the pip/packaging requirements. (We install pip==23.3.2 and packaging==23.2.)

To reproduce

pip install onnxruntime on a macos-11 runner will install 1.16.3.

pip install onnxruntime==1.17.0 on a macos-11 runner causes a crash.

Note: We pip install into a fresh Python 3.9 conda environment (pip+conda is a bit weird, so I wouldn't want to discount that as the potential cause). The packages that are installed when initializing the conda environment are:

 The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pip-23.3.1                 |   py39hecd8cb5_0         2.6 MB
    python-3.9.18              |       h5ee71fb_0        12.7 MB
    setuptools-68.2.2          |   py39hecd8cb5_0         950 KB
    tzdata-2023d               |       h04d1e81_0         117 KB
    wheel-0.41.2               |   py39hecd8cb5_0         109 KB
    ------------------------------------------------------------
                                           Total:        16.4 MB

The following NEW packages will be INSTALLED:

  ca-certificates    pkgs/main/osx-64::ca-certificates-2023.12.12-hecd8cb5_0 
  libcxx             pkgs/main/osx-64::libcxx-14.0.6-h9765a3e_0 
  libffi             pkgs/main/osx-64::libffi-3.4.4-hecd8cb5_0 
  ncurses            pkgs/main/osx-64::ncurses-6.4-hcec6c5f_0 
  openssl            pkgs/main/osx-64::openssl-3.0.12-hca72f7f_0 
  pip                pkgs/main/osx-64::pip-23.3.1-py39hecd8cb5_0 
  python             pkgs/main/osx-64::python-3.9.18-h5ee71fb_0 
  readline           pkgs/main/osx-64::readline-8.2-hca72f7f_0 
  setuptools         pkgs/main/osx-64::setuptools-68.2.2-py39hecd8cb5_0 
  sqlite             pkgs/main/osx-64::sqlite-3.41.2-h6c40b1e_0 
  tk                 pkgs/main/osx-64::tk-8.6.12-h5d9f67b_0 
  tzdata             pkgs/main/noarch::tzdata-2023d-h04d1e81_0 
  wheel              pkgs/main/osx-64::wheel-0.41.2-py39hecd8cb5_0 
  xz                 pkgs/main/osx-64::xz-5.4.5-h6c40b1e_0 
  zlib               pkgs/main/osx-64::zlib-1.2.13-h4dc903c_0 

Then, we explicitly upgrade pip in a separate step:

Requirement already satisfied: pip!=21.2.* in ./python/envs/venv_sct/lib/python3.9/site-packages (23.3.1)
Collecting pip!=21.2.*
  Using cached pip-23.3.2-py3-none-any.whl.metadata (3.5 kB)
Using cached pip-23.3.2-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.3.1
    Uninstalling pip-23.3.1:
      Successfully uninstalled pip-23.3.1
Successfully installed pip-23.3.2

Urgency

Not terribly urgent for us, but if this affects macOS more generally, then it is probably quite urgent!

Platform

Mac

EDIT: Silly bot! I chose Mac as my OS, but you've added the platform:windows tag. Oops.

OS Version

11

We also tested 12 and the same behavior persists. Presumably, this is also the case for 13/14?

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.17.0

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

[snnn]

I have difficulty on reproducing the issue.

I created a Github Action with macOS-11, then tried to install the latest ONNX Runtime version. It worked well.

https://github.com/snnn/ghtest/actions/runs/7749332259/job/21133675909

[joshuacwnewton]

Well, that's good to hear, at least. It probably means that it isn't urgent or widespread. :)

I tried to reproduce using a fresh miniconda environment, and there were no issues: https://github.com/joshuacwnewton/test-github-actions/actions/runs/7750152011/job/21135944774 EDIT: The miniconda environment was never activated, causing the wheel to get installed for the base environment, which is Python 3.12. This was not a proper reproduction, see below.

I can only assume that the problem lies the specific environment of the project I am working on, then. I will continue debugging on my end and report back when I find out what conditions this happens under.

[thorrester]

I'm noticing a similar issue. I can install in a macos env, but it leads to segmentation faults in macos github actions.
link

[joshuacwnewton]

I can install in a macos env, but it leads to segmentation faults in macos github actions. link

I think that may be a different issue? (One that would be better served by opening a separate issue.)

---

My issue is specifically related to pip not being able to see the newly released 1.17.0 under certain circumstances (which I am actively debugging at the moment to try and find reproducible steps for). This is similar to reports for other packages such as...

• Packages not discoverable by pip pypa/pip#10582

From the above issue, there is the following remark:

One possibility is the Conda Python is running on Rosetta and is therefore fetching x86_64 images instead. There are a lof of possibilities.

This seems like it might be on the right track, given the change from x64 wheels to universal2 wheels? Regardless, I'm getting the sense that this isn't an onnxruntime issue, and is actually more of a packaging/toolchain issue.

[joshuacwnewton]

Alright, I was able to reproduce the issue in an isolated workflow.

Running python -c "import platform; print(platform.platform())":

• Returns macOS-10.16-x86_64-i386-64bit after activating newly-created Miniconda environment.
• Returns macOS-11.7.10-x86_64-i386-64bit when the base Python is used.

10.16 and 11.0 seem to have once been synonyms for the same OS (beta vs. stable), so this looks like an upstream bug of some sort, whether in conda or in pip. In other words, the failure has more to do with the macOS version bump of the onnxruntime wheels (10_15 -> 11_0) than it does the universal2 wheel specifier. And, for some reason, activating a fresh miniconda environment causes the macOS version to be misdetected. Weird!!

Marking this issue as resolved for now. I will try to get to the bottom of how to fix this and update this comment later on, but I'll try not to bother the onnxruntime devs further with any more pings. :P

EDIT: With the problem now well-understood, I've found these related issues:

• pip on Python 3.8 won't install wheels that target macOS 11.0 or greater pypa/pip#11789

  macOS 10.16 doesn't exist, it's a compatibility shim that Apple put into the OS so that old software wouldn't get freaked out by the minor version going back to zero - macOS had been on version 10 for so long, apparently some software thought it would last forever and didn't bother checking it.

• Upgrade the vendored packaging to 22.0+ pypa/pip#11715

EDIT2: In the future, this line of debugging could be done much faster using the command python3 -m pip debug --verbose, which I found out about just now.

EDIT3: Looks like this is a dupe of:

• pip doesn't install latest version of onnxruntime #17166

That issue suggests the fix of SYSTEM_VERSION_COMPAT=0. And in my testing, I can confirm that this fixes the issue. 🎉