packaging does not know that macOS 10.16 and 11.0 are the same

[mkoeppe]

In python 3.9.9, as provided by the python.org macOS installer,

$ /Library/Frameworks/Python.framework/Versions/3.9/bin/python3
Python 3.9.9 (v3.9.9:ccb0e6a345, Nov 15 2021, 13:29:20) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.mac_ver()
('10.16', ('', '', ''), 'x86_64')

As a result, packaging 21.3 (and packaging main HEAD) do not include ...11_0... in packaging.tags.sys_tags().
However, wheel happily builds wheels with this tag, which pip (e.g. latest, 21.3.1) then refuses to install.

As platform.mac_ver() cannot be retroactively changed for released versions of Python, this should be addressed in packaging to make the support for macOS 11.x from #319 more robust.

Reference: https://trac.sagemath.org/ticket/33155

[pradyunsg]

Duplicate of #379?

[mkoeppe]

No

[mayeut]

packaging uses platform.mac_ver() as the only source to determine the current OS version.

packaging/packaging/tags.py

Line 356 in b10b90a

version_str, _, cpu_arch = platform.mac_ver()

There's definitely an issue here. Using the latest installers available on python.org for macOS:

Matt@MacBook-Pro ~ % python3.6 -c "import platform; print(platform.mac_ver())"
('10.16', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.7 -c "import platform; print(platform.mac_ver())"
('10.16', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.8 -c "import platform; print(platform.mac_ver())"
('10.16', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.9 -c "import platform; print(platform.mac_ver())"
('12.0.1', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.10 -c "import platform; print(platform.mac_ver())"
('12.0.1', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.6 -c "import platform; print(platform._mac_ver_xml())"
('10.16', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.7 -c "import platform; print(platform._mac_ver_xml())"
('10.16', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.8 -c "import platform; print(platform._mac_ver_xml())"
('10.16', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.9 -c "import platform; print(platform._mac_ver_xml())"
('12.0.1', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.10 -c "import platform; print(platform._mac_ver_xml())"
('12.0.1', ('', '', ''), 'x86_64')

The code for platform._mac_ver_xml() did not change much itself across versions.
It seems there are some tricks involved when reading /System/Library/CoreServices/SystemVersion.plist

10.16 and 11.0 are not the same and, at first glance, the only thing that seems to do the trick is to read this file from a subprocess:

Matt@MacBook-Pro ~ % python3.6 -c 'import subprocess, sys; subprocess.run([sys.executable, "-sS", "-c", "import platform; print(platform.mac_ver())"], env={"SYSTEM_VERSION_COMPAT": "0"})'
('12.0.1', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.7 -c 'import subprocess, sys; subprocess.run([sys.executable, "-sS", "-c", "import platform; print(platform.mac_ver())"], env={"SYSTEM_VERSION_COMPAT": "0"})'
('12.0.1', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.8 -c 'import subprocess, sys; subprocess.run([sys.executable, "-sS", "-c", "import platform; print(platform.mac_ver())"], env={"SYSTEM_VERSION_COMPAT": "0"})'
('12.0.1', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.9 -c 'import subprocess, sys; subprocess.run([sys.executable, "-sS", "-c", "import platform; print(platform.mac_ver())"], env={"SYSTEM_VERSION_COMPAT": "0"})'
('12.0.1', ('', '', ''), 'x86_64')
Matt@MacBook-Pro ~ % python3.10 -c 'import subprocess, sys; subprocess.run([sys.executable, "-sS", "-c", "import platform; print(platform.mac_ver())"], env={"SYSTEM_VERSION_COMPAT": "0"})'
('12.0.1', ('', '', ''), 'x86_64')

[mkoeppe]

10.16 and 11.0 are not the same

https://news.ycombinator.com/item?id=23910165

[mayeut]

When I said not the same, I meant 10.16 implies >= 11.0 rather than ==11.0
As can be seen above, I'm on macOS 12 but 10.16 is still reported so, in the context of this project, 10.16 shall not be considered "the same as" 11.0, it would just fix a subset of the issue. If I build a wheel for macOS 12, I want to be able to install it on macOS 12+ and that would not be the case if packaging replaces 10.16 with 11.0

[brettcannon]

Is there something more official from Apple's docs than a YC link to a blog post to a screenshot of Twitter for this policy? I'm not questioning that you're right, but it would be good to have something more official to link to if we add a hack to support this. I'm also interested in official docs to know if 10.16/11 is a one-off or if this will be a perpetual issue (i.e. there won't be a 10.17 for macOS 12).

And is there an equivalent issue on 'wheel' about there output differing from platform.mac_ver()?

[mkoeppe]

When I said not the same, I meant 10.16 implies >= 11.0 rather than ==11.0

OK, agreed.

[mkoeppe]

is there an equivalent issue on 'wheel' about there output differing from platform.mac_ver()?

wheel inspects the deployment targets of the built dynamic libraries to determine the tag - https://github.com/pypa/wheel/blob/main/src/wheel/macosx_libfile.py

[brettcannon]

And can someone include the complete list of tags they are seeing on Big Sur so we have a real-world example to work against?

[mayeut]

Is there something more official from Apple's docs than a YC link to a blog post to a screenshot of Twitter for this policy? I'm not questioning that you're right, but it would be good to have something more official to link to if we add a hack to support this. I'm also interested in official docs to know if 10.16/11 is a one-off or if this will be a perpetual issue (i.e. there won't be a 10.17 for macOS 12).

Agreed, I did not stumble upon official docs when (quickly) looking at this. Maybe @lawrence-danna-apple who wrote the PR
for #318 has some more official pointers to share ?

[mkoeppe]

is there an equivalent issue on 'wheel' about there output differing from platform.mac_ver()?

wheel inspects the deployment targets of the built dynamic libraries to determine the tag - https://github.com/pypa/wheel/blob/main/src/wheel/macosx_libfile.py

It boils down to this behavior:

>>> from wheel.macosx_libfile import calculate_macosx_platform_tag
>>> calculate_macosx_platform_tag('/usr/local/opt/gmp/lib', 'macosx-10.14-x86_64')  # from homebrew on macOS Big Sur 11.6 
[WARNING] This wheel needs a higher macOS version than the version your Python interpreter is compiled against.  To silence this warning, set MACOSX_DEPLOYMENT_TARGET to at least 11_0 or recreate these files with lower MACOSX_DEPLOYMENT_TARGET:  
/usr/local/opt/gmp/lib/libgmpxx.4.dylib
/usr/local/opt/gmp/lib/libgmp.dylib
/usr/local/opt/gmp/lib/libgmp.10.dylib
/usr/local/opt/gmp/lib/libgmpxx.dylib'macosx_11_0_x86_64'
>>> calculate_macosx_platform_tag('/Library/Frameworks/Python.framework/Versions/3.9/lib', 'macosx-10.14-x86_64')
'macosx_10_14_x86_64'
>>> calculate_macosx_platform_tag('/Library/Frameworks/Python.framework/Versions/3.9/lib', 'macosx-10.9-x86_64')
'macosx_10_9_x86_64'

[mayeut]

And can someone include the complete list of tags they are seeing on Big Sur so we have a real-world example to work against?

On macOS 12 (Monterey):
tags.python3.6.txt
tags.python3.10.txt

Dump done with the following command (packaging 5984e3b):

python -c $'from packaging.tags import mac_platforms\nfor tag in mac_platforms(): print(tag)' > tags.python3.10.txt

--- tags.python3.6	2022-01-13 22:08:21.000000000 +0100
+++ tags.python3.10	2022-01-13 22:08:37.000000000 +0100
@@ -1,3 +1,15 @@
+macosx_12_0_x86_64
+macosx_12_0_intel
+macosx_12_0_fat64
+macosx_12_0_fat32
+macosx_12_0_universal2
+macosx_12_0_universal
+macosx_11_0_x86_64
+macosx_11_0_intel
+macosx_11_0_fat64
+macosx_11_0_fat32
+macosx_11_0_universal2
+macosx_11_0_universal
 macosx_10_16_x86_64
 macosx_10_16_intel
 macosx_10_16_fat64

[mkoeppe]

tags.python3.9.bigsur.txt