Build on Apple Silicon

[hoechenberger]

This is currently blocked by actions/runner#805

EDIT: Now blocked by actions/runner-images#2187

Waiting on:

• Add arm64 migrations for openneuro-py, sleepecg, dcm2niix conda-forge/conda-forge-pinning-feedstock#2908 to be merged
• dcm2niix: waiting on ARM OSX Migrator + Updates conda-forge/dcm2niix-feedstock#17
• mne-kit-gui: waiting on mayavi to appear on CDN
• spyder: waiting on CDN
• fsleyes: waiting on ARM OSX Migrator conda-forge/fsleyes-feedstock#90
• sleepecg: waiting on CDNs
• mne-installer-menus: Fixed by bumping build number to release that doesn't depend on Spyder
• Add Mayavi and FSLeyes migration to osx-arm64 conda-forge/conda-forge-pinning-feedstock#2910
• openneuro-py
• get build working locally on an M1 machine
• get build cross-compile working locally on a non-M1 machine using something like:

  constructor --platform=osx-arm64 --conda-exe=path/to/conda-standalone
  

• add ARM to build matrix using command that worked locally
• merge PR
• cut a release
• test that the draft release installer actually works on M1

Status: https://conda-forge.org/status/#current_migrations

[larsoner]

Indeed automated builds are. For the 1.0 release, we should probably see how difficult it is to get an M1 laptop and run the builder manually there using your scripts. Even if it's 8h of work it seems worth it to me.

[hoechenberger]

I have access to an M1 machine, I just need to pick it up :) Running the build on it should be trivial. I just hope all dependencies are already available for Apple Silicon, but I'm quite confident that's the case!

[hoechenberger]

Unfortunately, qt hasn't been migrated to arm64 (Apple Silicon) yet on conda-forge:
conda-forge/qt-feedstock#217

So I guess we're stuck with Intel builds for now …

I'll pick up my Apple Silicon machine sometime this week and try out the installer with Intel packages to see whether and how this works (I have no idea if Rosetta 2 will just spin up transparently in the background, or how this is supposed to work)

@dengemann, what's your experience with non-native binaries on Apple Silicon? Can you just, like, click on them and they just run?

[hoechenberger]

I've opened a PR to migrate MNE and all dependencies to arm64, which will allow us to keep track of the process via https://conda-forge.org/status/#current_migrations

[hoechenberger]

Besides the delays on the conda-forge side of things, GH Actions runners are still not ready to run on Apple Silicon natively either. Should be in a few weeks though:
actions/runner#805 (comment)

[dengemann]

@dengemann, what's your experience with non-native binaries on Apple Silicon? Can you just, like, click on them and they just run?

Yep that's it. They just work. Although native is always much better.

[hoechenberger]

Intel installers & MNE now work as expected (through Rosetta 2) on Apple Silicon thanks to #73
If Rosetta is not yet installed, the installer will prompt the user to install it (it's just a single click)

First startup takes quite a bit of time, but after that everything works.

Still waiting for ARM builds of Qt.

[larsoner]

A pre-release version of the macOS-ARM64 runner is available now in runner version 2.292.0 or later

@hoechenberger I have access to an M1 mac, so I could do these builds manually the 5 or so times we'll need it again until actions/runner-images#2187 is sorted out (now that GH actions isn't the bottleneck anymore: actions/runner#805 (comment)). I also just set up an environment on my M1 machine and everything I needed seemed to be available, but I did install PyQt6 from pip because I was doing things piecemeal, so we'll see.

I think at some point @hoechenberger you sent instructions for how to create an installer locally. I'll look for those and try it locally on my Linux machine, and if it produces something usable, I'll try my M1 machine using the same script/instructions...

[hoechenberger]

@larsoner I have been using an M1 Mac myself for a few months now, too :)

We still don't have qt for M1 … there is qt-main now but I'm not sure if it does the trick. And yes, no Qt6 builds anywhere to be seen …

[hoechenberger]

It seems I was mistaken an pyqt already depends on qt-main, meaning we should have everything we need for running natively on M1. I'm just wondering why the migration bot hasn't caught up with mne-feedstock yet … I'll take a look.

https://conda-forge.org/docs/user/announcements.html#id1

[hoechenberger]

I have an x84 conda install on my M1 Mac, and the following commands set up a native Apple Silicon environment:

CONDA_SUBDIR=osx-arm64 mamba create -n mne-base-M1 mne-base
CONDA_SUBDIR=osx-arm64 mamba create -n mne-M1 mne

but the 2nd command wants to install an outdated version of mne-qt-browser. I hope this will be resolved once conda-forge/mne-feedstock#97 has been merged. I will also add mne-qt-browser to the list of arm64 migrations at https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/migrations/osx_arm64.txt

cc @marsipu

[hoechenberger]

I've requested the mne-qt-browser migration to osx-arm64 (Apple Silicon)

conda-forge/conda-forge-pinning-feedstock#2905

[hoechenberger]

Both PRs have been merged (dropping spyder dependency from mne feedstock, and adding mne-qt-browser to the list of requested migrations to osx-arm64).

This means that hopefully within the next few hours, some changes should become visible at https://conda-forge.org/status/#armosxaddition, and a bot should open a migration PR at https://github.com/conda-forge/mne-qt-browser-feedstock

Fingers crossed!

[larsoner]

It looks like qt is in-PR and mne-qt-browser is still awaiting-PR 6 hours later. I guess it's not surprising Qt is taking a while, it's probably a big compile, and might have issues.

[hoechenberger]

No, qt is not going to be built anytime soon and we don't need it -- it will be turned into a meta-package.

We only need qt-main, on which pyqt depends -- and they're both available. I asked on the conda-forge Gitter channel and they told me that the "awaiting-pr" label means that the bot is scheduled to make the PR (all deps are fulfilled), but that rate limiting causes a wait. It should happen within the next hours or day or so :)

[larsoner]

Ahh okay great!

[hoechenberger]

Ok cool, so now with all MNE dependencies migrated to arm64, we should check if any of the numerous other things we include in the installer are still blocked

I think Spyder is still having issues, because qtconsole is awaiting-pr:
https://conda-forge.org/status/#armosxaddition

Let's just wait until tomorrow or so, I expect Spyder to become available for arm64 by then :)

[hoechenberger]

update: migration for qtconsole currently fails :(

conda-forge/qtconsole-feedstock#48

[larsoner]

And also locally on M1 I get:

$ conda env create -f environment.yml 
Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound: 
  - constructor==3.3a[build=*_21]

So that needs to be migrated, too, apparently? It seems like I can conda install -c conda-forge constructor, so it seems to be a problem with the forked version. @hoechenberger can you look into getting this forked version of constructor on arm as well?

[hoechenberger]

Oh dang, yes I suppose it was never migrated. I can reach out to the person who created the fork.

[hoechenberger]

I think a few of the changes from the fork have also been merged upstream, I can look into this and see if we still need to rely on that fork or if we're ready to move back to upstream vanilla

[hoechenberger]

I dropped the author a message.

In the worst case, we can just build those packages on Apple Silicon ourselves and upload them to a custom channel on anaconda.org!

[larsoner]

At least commenting out the versions in environment.yml allowed conda env create -f environment.yml to succeed on M1, then commenting out welcome_file, readme_text, conclusion_file, notarization_identity_name, reverse_domain_identifier in construct.yaml allowed it to try. Now the packages that fail are:

conda.exceptions.ResolvePackageNotFound: 
  - sleepecg
  - spyder
  - openneuro-py
  - dcm2niix

@hoechenberger can you look into migrating or pinging people for sleepecg, openneuro-py, and dcm2niix?

[hoechenberger]

At least commenting out the versions in environment.yml allowed conda env create -f environment.yml to succeed on M1

Yes this then should cause you to use the vanilla packages from conda-forge, which are lacking some of the stuff we need. But good idea as this allows us to see which other packages are problematic!!

@hoechenberger can you look into migrating or pinging people for sleepecg, openneuro-py, and dcm2niix?

I believe I'm responsible for all of them cough :D Edit except for dcm2niix

When I start touching those feedstocks, may I add you as co-maintainer..?

[hoechenberger]

@larsoner I believe sleepecg is blocked by tensorflow…

[larsoner]

Okay, so maybe for now we add a platform selector for sleepecg that excludes it on M1, but hopefully that'll be the only one for now. I think eventually we're going to have the same problem eventually for mne-iclabel.

Yes feel free to add me as a co-maintainer

[hoechenberger]

@larsoner I've requested the migrations via conda-forge/conda-forge-pinning-feedstock#2908

So even if we still have some blockers, these packages will show up on the migration status page

[larsoner]

Awesome. And am I interpreting things right that Spyder just needs qtconsole, so now that its PR is no longer stalled, it should be built soon hopefully as well?

[hoechenberger]

And am I interpreting things right that Spyder just needs qtconsole, so now that its PR is no longer stalled, it should be built soon hopefully as well?

Yes, that's my current understanding!

[hoechenberger]

Okay, so maybe for now we add a platform selector for sleepecg that excludes it on M1, but hopefully that'll be the only one for now. I think eventually we're going to have the same problem eventually for mne-iclabel.

I just looked and tensorflow is actually already available for arm64 on conda-forge!

https://anaconda.org/conda-forge/tensorflow-base

[larsoner]

Okay, can you migrate sleepecg then?

[hoechenberger]

Okay, can you migrate sleepecg then?

This should happen automatically once conda-forge/conda-forge-pinning-feedstock#2908 has been merged! Let's hope for the best!

[hoechenberger]

I dropped the author a message.

They got back to me with good news! We don't need arm64 builds of the constructor, as it can cross-build. This is what they wrote me:

Essentially, make sure you do have a conda-standalone for osx-arm64 (available in our napari channel) and pass its path like constructor --platform=osx-arm64 --conda-exe=path/to/conda-standalone

So I think what this means is that if we manually extract the conda-standalone package and pass the path to the binary via the conda-exe switch, we should be able to build the Apple Silicon installers even on a different platform!

[larsoner]

That's awesome and actually makes some sense. Let's get #125 in and then we can locally test and iterate on this idea locally on non-M1 machines (once the builds work on M1 machines natively, see TODO list I added at the top)

[hoechenberger]

Oh and for installing the deps, when you create the environment for the forked constructor on your Mac, install only Intel binaries via

CONDA_SUBDIR=osx-64 mamba create ...

[hoechenberger]

@larsoner There is still an entire chain of macOS dependencies (pyobjc-framework-*) that need to be migrated to the latest macOS SDK for Spyder, the blocker being pyobjc-framework-cocoa if I'm not mistaken. Let's hope this will be resolved today or within the next few days.

[hoechenberger]

@larsoner I believe once conda-forge/pyobjc-framework-cocoa-feedstock#54 has been merged, migration of the other packages should follow 🙏

Dependency graph, but I believe it's pyobjc-framework-cocoa blocking pyobjc-framework-fsevents:

[hoechenberger]

I've made a PR to hopefully get the pyobjc-framework-fsevents migration working again:
conda-forge/pyobjc-framework-fsevents-feedstock#22

[hoechenberger]

@larsoner What should we do re conda-forge/pyobjc-framework-fsevents-feedstock#22 ? Should we try to ping the maintainer again? Or would that be too early?

[larsoner]

I think we just wait until Monday :(

[hoechenberger]

@larsoner Looking at their GH activity, I assume they're simply on vacation, which makes sense since yesterday was a public holiday and lots of people are taking the opportunity to have a super-long weekend. So we'll just have to be patient and let this person rest 👍👍👍

[larsoner]

Looks like someone else did the pinging for you, which is nice :) Hopefully it gets merged today!

[larsoner]

@hoechenberger do we need to do something to get pyobjc-framework-coreservices out of the "error" in https://conda-forge.org/status/#current_migrations now that conda-forge/pyobjc-framework-fsevents-feedstock#22 has been merged?

[hoechenberger]

@hoechenberger do we need to do something to get pyobjc-framework-coreservices out of the "error" in https://conda-forge.org/status/#current_migrations now that conda-forge/pyobjc-framework-fsevents-feedstock#22 has been merged?

I'm not sure, let's wait until ...-fsevents becomes available from the CDNs and then make a PR for a manual migration instead of waiting on the bot; see this PR for an example of how to do it (look at the individual commits):

conda-forge/sleepecg-feedstock#12

[larsoner]

Actually it looks like that's no longer the blocker but rather pyobjc-framework-cocoa being "in PR" ... despite the plan appearing to be not merging that PR. Not sure how to get the conda-forge status to understand the situation

[larsoner]

because we have

• spyder, waiting on
• pyobjc-framework-coreservices, waiting on
• pyobjc-framework-fsevents, waiting on
• pyobjc-framework-cocoa

And separately we also have spyder waiting on some other stuff (qtconsole, pyqt, qt-webengine, qt-main)

[hoechenberger]

Actually it looks like that's no longer the blocker but rather pyobjc-framework-cocoa being "in PR" ... despite the plan appearing to be not merging that PR. Not sure how to get the conda-forge status to understand the situation

No this is another migration for linux-aarch64 and ppc64le, so unrelated to osx-arm64

[larsoner]

Argh right, clicking the wrong buttons!

[hoechenberger]

That's the dependency graph I'm getting for Spyder

[hoechenberger]

@larsoner Only now the ...-fsevents feedstock is actually being migrated:
conda-forge/pyobjc-framework-fsevents-feedstock#23

[hoechenberger]

… still waiting for the PR to get merged … sigh

[hoechenberger]

Merged!

We're almost there:

[hoechenberger]

@larsoner The finish line is in sight!!!

conda-forge/spyder-feedstock#133