Fix support for Apple M1 #753

juj · 2021-03-10T14:57:39Z

Fix support for Apple M1. Node.js will still run via Rosetta 2 emulation since they do not yet have M1 support, but Python, LLVM, Emscripten and Binaryen will be native.

In order for Python to actually be native, will need to upload new Python to Google CDN built with update_python.py.

…ion since they do not yet have M1 support, but Python, LLVM, Emscripten and Binaryen will be native.

sbc100

Generally looks great. If you want to attach the python tar archive I'm happy to upload it to google storage for you.

emsdk_manifest.json

scripts/update_python.py

.gitignore

juj · 2021-03-11T10:30:06Z

macOS Arm64 python: http://clb.confined.space/dump/python-3.9.2-1-macos-arm64.tar.gz

sbc100

lgtm!

arm64 python uploaded.

Do you want me to upload the x86-64 mac python too or (perhaps better) leave that version as if for now (will require forking upstream-main on line 490 I guess?)

juj · 2021-03-11T18:05:36Z

Our min spec for Intel macOS is version 10.13, so I've been wrangling today a chance to do the Intel side build either on real macOS 10.13, or with environment variables MACOSX_DEPLOYMENT_TARGET=10.13 SDKROOT=/Developer/SDKs/MacOSX10.13.sdk and

./configure CFLAGS="-mmacosx-version-min=10.13 -isysroot /Developer/SDKs/MacOSX10.13.sdk" \
            CXXFLAGS="-mmacosx-version-min=10.13 -isysroot /Developer/SDKs/MacOSX10.13.sdk" \
            LDFLAGS="-mmacosx-version-min=10.13 -isysroot /Developer/SDKs/MacOSX10.13.sdk"

(where /SDKs/MacOSX10.13.sdk should be replaced with whatever SDK path one actually has installed), but I wasn't able to get that going today.

If you'd like to do an upload, if you can do it with the above flags, that'd be great. Or you can do just an upload with a regular build, and I can also do a local build just for our Unity deployment purposes. Or if you want to wait, I can submit that build as well once I get my hands on a suitable Intel mac environment to work with.

sbc100 · 2021-03-11T18:17:10Z

Can we just least the x86-64 mac python on 3.7.4 and not upload anything new there?

sbc100 · 2021-03-11T18:17:55Z

Can we just least the x86-64 mac python on 3.7.4 and not upload anything new there?

This would also make this change more precise as it will effect M1 users.

juj · 2021-03-11T18:43:11Z

Can we just least the x86-64 mac python on 3.7.4 and not upload anything new there?

We could, but it will require splitting the manifest sdks part per both macOS archs. I would prefer not to do that.

sbc100 · 2021-03-11T19:03:15Z

Can we just least the x86-64 mac python on 3.7.4 and not upload anything new there?

We could, but it will require splitting the manifest sdks part per both macOS archs. I would prefer not to do that.

I don't really mind which option we choose, but this PR is currently blocked on getting a new python x86-64 binary right? Forking the manifest is one way to unblock it. I don't have access to a macOS machine myself. (I guess its possible to use the circleci mac via SSH bot perhaps?)

juj · 2021-03-11T19:32:25Z

this PR is currently blocked on getting a new python x86-64 binary right?

That's right - I'll unblock that tomorrow and upload the new binary, did not have a chance to do it today.

juj · 2021-03-12T11:42:33Z

Added http://clb.confined.space/dump/python-3.9.2-1-macos-x86_64.tar.gz for macOS 10.13 min version build.

Also can you reupload http://clb.confined.space/dump/python-3.9.2-1-macos-arm64.tar.gz . I updated that one to run with minimum version 11.0 (rather than 11.1 that it was previously targeting).

Also updated the PR to specify these 10.13 and 11.0 minimum requirements for the build script.

sbc100 · 2021-03-12T15:27:32Z

Added http://clb.confined.space/dump/python-3.9.2-1-macos-x86_64.tar.gz for macOS 10.13 min version build.

Also can you reupload http://clb.confined.space/dump/python-3.9.2-1-macos-arm64.tar.gz . I updated that one to run with minimum version 11.0 (rather than 11.1 that it was previously targeting).

Also updated the PR to specify these 10.13 and 11.0 minimum requirements for the build script.

Done

What was the min version in the previous/current version of python for x86_64?

juj · 2021-03-12T15:29:51Z

What was the min version in the previous/current version of python for x86_64?

It would be whatever macOS version was installed on the system where the zips were created on.

sbc100 · 2021-03-12T15:31:45Z

scripts/update_python.py

+    min_macos_version_line = '-mmacosx-version-min=' + min_macos_version  # Specify the min OS version we want the build to work on
+    build_flags = min_macos_version_line + ' -Werror=partial-availability'  # Build against latest SDK, but issue an error if using any API that would not work on the min OS version
+    env = os.environ.copy()
+    env['MACOSX_DEPLOYMENT_TARGET'] = min_macos_version


Apparently you only need either the command line of the environment variable, not both:

https://stackoverflow.com/questions/25352389/what-is-the-difference-between-macosx-deployment-target-and-mmacosx-version-min

That is true, but there is ambiguity in which method is preferred (https://stackoverflow.com/questions/25352389/what-is-the-difference-between-macosx-deployment-target-and-mmacosx-version-min highlights the cmdline flag no longer works, and https://github.com/devernay/xcodelegacy#using-the-older-sdks highlights the environment variable should be deprecated), and which method is supported on different compilers, so the general practice is to supply both. There is no harm in specifying them both for the build.

sbc100 · 2021-03-12T15:32:57Z

scripts/update_python.py

+            min_macos_version = '10.13'
+        elif platform.machine() == 'arm64':
+            prefix = '/opt/homebrew'
+            min_macos_version = '11.0'


else assert?

the code will fail on the next line with these variables not defined, preferred not to add redundant lines (like the unnecessary else: in if x: return foo else: you often comment about)

sbc100 · 2021-03-12T15:37:42Z

What was the min version in the previous/current version of python for x86_64?

It would be whatever macOS version was installed on the system where the zips were created on.

I wonder if we can figure that out.. it would be good to know if we are increasing of decreasing the min requirments with this change.. maybe we inspect the binaries to figure it out (i'll give it a go)

sbc100 · 2021-03-12T15:39:03Z

Perhaps should also mention this in the Mac OS X / System Requirements section of the README.md?

sbc100 · 2021-03-12T15:43:24Z

Looks like the previous version was 10.12:

$ llvm-objdump -p python3
Load command 9
      cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.12
      sdk 10.12

juj · 2021-03-13T08:59:07Z

macOS 10.13 was released in September 2017. Cross comparing system requirements at https://en.wikipedia.org/wiki/MacOS_High_Sierra vs https://en.wikipedia.org/wiki/MacOS_Sierra , there are no mac hardware that would not be able to update from macOS 10.12 to macOS 10.13 (and hardware from 2009 onwards is compatible), so not concerned about bumping the minimum version.

Emsdk on macOS requires system installed python in order to install Emsdk bundled python.

According to https://medium.com/@youngstone89/mac-os-x-sierra-python-installation-28db734a2a60 and https://stackoverflow.com/questions/41647424/how-to-install-updated-python-on-mac-os-sierra-with-brew, macOS 10.12 had Python 2.7.10 preinstalled, no Python 3.

According to https://stackoverflow.com/a/33175893 and https://stackoverflow.com/questions/51804203/atom-prioritizes-python-2-7-over-python-3-7-on-macos macOS 10.13 ships with the same Python 2.7.10 as well. So that will be the same minimum version that Emsdk needs to work with.

sbc100 · 2021-03-13T16:52:57Z

macOS 10.13 was released in September 2017. Cross comparing system requirements at https://en.wikipedia.org/wiki/MacOS_High_Sierra vs https://en.wikipedia.org/wiki/MacOS_Sierra , there are no mac hardware that would not be able to update from macOS 10.12 to macOS 10.13 (and hardware from 2009 onwards is compatible), so not concerned about bumping the minimum version.

Emsdk on macOS requires system installed python in order to install Emsdk bundled python.

According to https://medium.com/@youngstone89/mac-os-x-sierra-python-installation-28db734a2a60 and https://stackoverflow.com/questions/41647424/how-to-install-updated-python-on-mac-os-sierra-with-brew, macOS 10.12 had Python 2.7.10 preinstalled, no Python 3.

According to https://stackoverflow.com/a/33175893 and https://stackoverflow.com/questions/51804203/atom-prioritizes-python-2-7-over-python-3-7-on-macos macOS 10.13 ships with the same Python 2.7.10 as well. So that will be the same minimum version that Emsdk needs to work with.

This is all fine with me. I just want to make sure we track this stuff.

Thanks for investigating.

juj requested a review from sbc100 March 10, 2021 14:57

juj force-pushed the apple_m1_support branch from 30134e4 to e38fbf8 Compare March 10, 2021 15:01

Fix support for Apple M1. Node.js will still run via Rosetta 2 emulat…

3cda2f5

…ion since they do not yet have M1 support, but Python, LLVM, Emscripten and Binaryen will be native.

juj force-pushed the apple_m1_support branch from e38fbf8 to 3cda2f5 Compare March 10, 2021 15:10

sbc100 reviewed Mar 10, 2021

View reviewed changes

emsdk_manifest.json Outdated Show resolved Hide resolved

scripts/update_python.py Show resolved Hide resolved

.gitignore Outdated Show resolved Hide resolved

.gitignore Outdated Show resolved Hide resolved

juj added 2 commits March 11, 2021 12:26

Update M1 python version and URL

2d67ac5

Merge remote-tracking branch 'origin/master' into apple_m1_support

fba0fd2

juj added 2 commits March 11, 2021 14:05

Remove .gitignore additions

553b693

Move python first in the manifest (#441)

b0014eb

sbc100 approved these changes Mar 11, 2021

View reviewed changes

Use macosx-version-min when building python

35b1b0c

Update Intel macOS python package name

acc0a50

juj merged commit 96d7ee1 into master Mar 12, 2021

juj deleted the apple_m1_support branch March 12, 2021 15:29

sbc100 reviewed Mar 12, 2021

View reviewed changes

sbc100 approved these changes Mar 12, 2021

View reviewed changes

juj mentioned this pull request Mar 15, 2021

installing emsdk on Apple M1 hardware fails #671

Closed

sbc100 mentioned this pull request Mar 22, 2021

Quick hack to treat macOS-arm64 as macOS-x86_64 #771

Open

sbc100 mentioned this pull request Feb 22, 2022

incorrect node on non-x86/x86_64 systems #72

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix support for Apple M1 #753

Fix support for Apple M1 #753

juj commented Mar 10, 2021

sbc100 left a comment

juj commented Mar 11, 2021

sbc100 left a comment

juj commented Mar 11, 2021

sbc100 commented Mar 11, 2021

sbc100 commented Mar 11, 2021

juj commented Mar 11, 2021

sbc100 commented Mar 11, 2021

juj commented Mar 11, 2021

juj commented Mar 12, 2021

sbc100 commented Mar 12, 2021

juj commented Mar 12, 2021

sbc100 Mar 12, 2021

juj Mar 13, 2021

sbc100 Mar 12, 2021

juj Mar 13, 2021

sbc100 commented Mar 12, 2021

sbc100 commented Mar 12, 2021

sbc100 commented Mar 12, 2021

juj commented Mar 13, 2021

sbc100 commented Mar 13, 2021

Fix support for Apple M1 #753

Fix support for Apple M1 #753

Conversation

juj commented Mar 10, 2021

sbc100 left a comment

Choose a reason for hiding this comment

juj commented Mar 11, 2021

sbc100 left a comment

Choose a reason for hiding this comment

juj commented Mar 11, 2021

sbc100 commented Mar 11, 2021

sbc100 commented Mar 11, 2021

juj commented Mar 11, 2021

sbc100 commented Mar 11, 2021

juj commented Mar 11, 2021

juj commented Mar 12, 2021

sbc100 commented Mar 12, 2021

juj commented Mar 12, 2021

sbc100 Mar 12, 2021

Choose a reason for hiding this comment

juj Mar 13, 2021

Choose a reason for hiding this comment

sbc100 Mar 12, 2021

Choose a reason for hiding this comment

juj Mar 13, 2021

Choose a reason for hiding this comment

sbc100 commented Mar 12, 2021

sbc100 commented Mar 12, 2021

sbc100 commented Mar 12, 2021

juj commented Mar 13, 2021

sbc100 commented Mar 13, 2021