Switch to arm64 version of python on arm64 macOS (M1)

[sbc100]

Currently we are using the x86_64 (emulated) version of python on arm64 macOS which then detects that architecture as x86_64 and downloads the x86_64 binaries even though we have usable arm binaries since #816.

See: emscripten-core/emscripten#14031

[sbc100]

@juj WDYT about doing this?

[sbc100]

@dschuff do you still have access to an M1 mac to try and do this?

[juj]

M1 should use arm64 version of python - it even ships with an arm64 version of python built-in. When I did the initial M1 support, I did test that if bootstrapping using a M1 version of python, emsdk will download/build a M1 version of python as well.

But there are lots of x64 python installations that are available for Macs. I think the issue is that if one runs emsdk under a x64 emulated python for initial bootstrapping, then emsdk will think that the system is a x64 one and not a M1 one, and will then proceed to installing x64 versions of all the tools in the toolchain. (running x64 python under rosetta will have python advertise to the user code that the system is a x64 system even if it is in reality arm-based)

[ragfield]

I tried uninstalling Emcripten's x86_64 version of python3:

schofield@RobBookPro13 emsdk % ./emsdk uninstall python-3.9.2-1-64bit
Uninstalling tool 'python-3.9.2-1-64bit'..
Deleting path '/usr/local/opt/emsdk/python/3.9.2-1_64bit'
Done uninstalling 'python-3.9.2-1-64bit'.

But it was immediately reinstalled by emsdk:

schofield@RobBookPro13 emsdk % ./emsdk install latest
Resolving SDK alias 'latest' to '2.0.32'
Resolving SDK version '2.0.32' to 'sdk-releases-upstream-74646397e3c5010824ad60d1de86c6bcbe334dff-64bit'
Installing SDK 'sdk-releases-upstream-74646397e3c5010824ad60d1de86c6bcbe334dff-64bit'..
Skipped installing node-14.15.5-64bit, already installed.
Installing tool 'python-3.9.2-1-64bit'..
Downloading: /usr/local/opt/emsdk/zips/python-3.9.2-1-macos-x86_64.tar.gz from https://storage.googleapis.com/webassembly/emscripten-releases-builds/deps/python-3.9.2-1-macos-x86_64.tar.gz, 32022000 Bytes
Unpacking '/usr/local/opt/emsdk/zips/python-3.9.2-1-macos-x86_64.tar.gz' to '/usr/local/opt/emsdk/python/3.9.2-1_64bit'
Done installing tool 'python-3.9.2-1-64bit'.
Skipped installing releases-upstream-74646397e3c5010824ad60d1de86c6bcbe334dff-64bit, already installed.
Done installing SDK 'sdk-releases-upstream-74646397e3c5010824ad60d1de86c6bcbe334dff-64bit'.

[sbc100]

Can you do that again and see which version of python was used for your installation?

I think running with sh -x should show you this. e.g: sh -x ./emsdk install latest

[sbc100]

Indeed, from looking at the manifest it looks like if you bootstrap with an arm64 version of python emsdk will install python-3.9.2-1-macos-arm64.tar.gz and not python-3.9.2-1-macos-x86_64.tar.gz.. so it looks like the bootstrapping python is wrong here.

Perhaps we can/should switch to a different method of arch detection to avoid this issue?

[juj]

I tried uninstalling Emcripten's x86_64 version of python3:

But it was immediately reinstalled by emsdk:

I believe that is because the python you that ended up being run with emsdk was a Rosetta x64 emulated one, hence it would take this path:

emsdk/emsdk.py

Line 118 in 2b3e0b9

ARCH = 'x86_64'

If the executing python was a arm64 one, it should take this path:

emsdk/emsdk.py

Line 122 in 2b3e0b9

ARCH = 'aarch64'

Perhaps we can/should switch to a different method of arch detection to avoid this issue?

I hesitated to change that on the first run, anticipating that maybe someone who installs x64 versions of python would actually have a reason for that and wanted to be running x64 variants deliberately (to avoid a bug?). Though maybe that does not make sense.

We should be able to detect arm64 host from within python by running uname -m process and grabbing the stdout and checking for arm64 string there.

[juj]

On a clean/stock install of a M1 Mac Mini, I did observe that the system python was native arm64, and then subsequently emsdk installing there would get a full M1 version of all tools after the PR that added arm64 variant of python. Back when authoring that, there didn't exist official arm64 version of node.js (so node would be the only tool that'd remain under rosetta), though that may have changed by now.

Maybe users have e.g. homebrew or macports or other installations of python that have downloaded a x64 version. Or some other source?

[ragfield]

Can you do that again and see which version of python was used for your installation?

It looks like I had a very old version (so old it wasn't listed unless I ran ./emsdk list --old) of python in emsdk/python. Actually, I had a lot of old stuff in various emsdk subdirectories. I deleted my emsdk directory, re-cloned the emsdk repo, re-installed latest. Now I have an arm64 clang, though curiously it claims its target is x86_64:

schofield@RobBookPro13 emsdk % file upstream/bin/clang          
upstream/bin/clang: Mach-O 64-bit executable arm64
schofield@RobBookPro13 emsdk % upstream/bin/clang --version
clang version 14.0.0 (https://github.com/llvm/llvm-project 9403514e764950a0dfcd627fc90c73432314bced)
Target: x86_64-apple-darwin21.1.0
Thread model: posix
InstalledDir: /usr/local/opt/emsdk/upstream/bin

I don't know whether this is related to the target anomaly, but the -Wpoison-system-directories problem from emscripten-core/emscripten#14031 is still present. I'll update that issue accordingly.

It appears the architecture problems I've been facing are entirely caused by upgrading from older versions and won't affect new installations.

The node version installed by emsdk is still x86_64. I do have a homebrew version of node that is arm64 native.

[sbc100]

OK, I think I will close this for now. We could switch to using a python-process-external way of detecting the arch.. but it seems like this would only effect some folks and there is a work around so its not urgent.