Universal2 wheels on macOS

[joerick]

Add support for building arm64 and universal2 wheels on macOS 11+.

• Building arm64 and universal2 on x86_64 runners using CIBW_ARCHS
• Testing universal2 wheels
• Docs update
  • CIBW_ARCHS option
  • 'How to' build for Apple Silicon, (probably including the nuances around testing)
• Fix up CIBW_TEST_SKIP support for specifying parts of universal2 wheels
• Examples update?

Closes #473.

[joerick]

Hmm, due to actions/runner-images#2247 this is gonna be slow going... it might be better to start work on this next week...

[henryiii]

We will have to download the new images for Python 3.9 specifically https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg (which, regardless of the name, does support 10.9+ and Intel too). I think 3.8 may get a release as well.

I think the builds are a bit slower, but still faster than Travis!

[henryiii]

Do the 10.6 32/64 fat binaries not work on macOS 11? Is that perhaps the issue with 3.5? 2.7 does seem to be working, but not 3.5, if I read the log correctly? Possibly it's the workaround we have in place for SSL?

Just a thought, how about we drop 3.5 on Windows and macOS, but keep it on Linux for a little while longer (April 2021, EOL for Ubuntu 16.04, or whenever manylinux wheels drop it - ideally, I think we should match exactly what they are doing on Linux)

[Czaki]

Do the 10.6 32/64 fat binaries not work on macOS 11? Is that perhaps the issue with 3.5? 2.7 does seem to be working, but not 3.5, if I read the log correctly?

https://discourse.jupyter.org/t/jupyter-notebook-error-dyld-library-not-loaded-corefoundation-after-macos-big-sur-upgrade/6838
https://stackoverflow.com/questions/64916693/jupyter-notebook-error-dyld-library-not-loaded-corefoundation-after-macos-big

So it looks like the bigger problem with all not patched python versions..

[henryiii]

I'm not sure. It's not affecting 2.7, and Big Sur comes with 3.8.2 (CLT), so I think it's worth checking to see if 3.6 builds, anyway - maybe just reverse the order and do 3.9 first, then 3.8, etc. :) It might be that the Universal2 3.9 wheel will need to be in a separate job for now, anyway - but that shouldn't stop us from working out the 11.0 issues. @joerick let me know if / when it's okay to push to this branch. It's better if you do it for now, as I have to prepare to record some CMake training videos, but I'd like to help this along when you aren't working on it.

[joerick]

We will have to download the new images for Python 3.9 specifically https://www.python.org/ftp/python/3.9.1/python-3.9.1-macos11.0.pkg (which, regardless of the name, does support 10.9+ and Intel too). I think 3.8 may get a release as well.

I think the builds are a bit slower, but still faster than Travis!

Hm, I wonder if you're talking about native ARM support, or the ability to build universal2 wheels? I was under the impression that wheel and the Xcode toolchain was cross-compiling both x86_64 and arm by default, regardless of the Python version?

I'll disable the 3.5 macOS build for now. Feel free to hack on this branch @henryiii, it just takes about half an hour for each macOS build to queue so it's not a lot of fun :)

[henryiii]

PyPy is not macOS 11 ready either? At least all the normal wheels built. Though are they univesal2? Don't see a difference in naming.

[joerick]

Building wheel for spam (setup.py): started
316
    Building wheel for spam (setup.py): finished with status 'done'
317
    Created wheel for spam: filename=spam-0.1.0-cp37-cp37m-macosx_10_9_x86_64.whl size=2382 sha256=efb4f386693223f8a943c5b60d7571318f0258eda5a7339992c62e957e40d493
318
    Stored in directory: 

Yeah, maybe you're right that we need the new Python macos11 builds

[Czaki]

PyPy is not macOS 11 ready either? At least all the normal wheels built. Though are they univesal2? Don't see a difference in naming.

In the current release, of the wheel depends on get_platform from distutils.util.
Its output depends on the installer (it is hardcoded, not calculated).

So without changing the installer platform tag will be x86.

As I do not understand, there is no plan for changing this behavior.

[henryiii]

...built_wheel/spam-0.1.0-cp39-cp39-macosx_10_9_universal2.whl ...

Looks like that did the trick. :) How do we want to switch to using the new download? Always, with a flag, etc? Can you build an Intel (or Apple Silicon) wheel from the universal Python binary? If so, I don't see a reason to not always download it?

Also, PyPy and Python 3.5 don't support macOS 11, but I think that's okay - we could either recommend they be skipped for now until 11.0 becomes the default for macOS-latest on one of the CI systems, and by then, hopefully PyPy will have fixed their issues. @mattip is this a known issue with PyPy? Also will it be able to make Universal2 wheels too eventually? I feel like I have seen this in homebrew too for the 11.0 (but could be mixing it with Apple Silicon support - that I know is missing currently for PyPy). GitHub is pretty fast, so maybe macOS-latest will change sooner than I think, but I bet we have some time.

[henryiii]

More wheel related, if you need a really new pip to use a universal2 wheel, I expect most projects will provide both for the near future - in which case, it seems much more sensible to provide a Intel and an Apple Silicon wheel instead of an Intel and a Universal2 wheel. So the question remains, can you set a target?

[henryiii]

Also, can we silence all the silly warnings from pip? The Python 2 warning, the newer version available warning? We pin pip, so these warnings just get in the way when trying to find the real error.

[henryiii]

And, final message tonight - do we have any 10.x builds left? Azure and GHA ran on 11, and I can't see CircleCI, so can't tell if this still works on 10.15 or if using the new Python broke it.

[mattip]

Also, PyPy and Python 3.5 don't support macOS 11

Is that this issue with _ctypes and SCDynamicStoreCopyProxies or are there more problems?

Edit: this issue

[Czaki]

Looks like that did the trick. :) How do we want to switch to using the new download? Always, with a flag, etc? Can you build an Intel (or Apple Silicon) wheel from the universal Python binary? If so, I don't see a reason to not always download it?

I do not know the method to do this. As far as I know, there is a need to check the Xcode version. Maybe @ronaldoussoren will know more.

Maybe the best workaround will be updated delocate.

[joerick]

perhaps you quoted the wrong text there, @Czaki ?

Also, can we silence all the silly warnings from pip? The Python 2 warning, the newer version available warning? We pin pip, so these warnings just get in the way when trying to find the real error.

If we can be selective enough about the warnings, yes please :)

I can't see CircleCI, so can't tell if this still works on 10.15 or if using the new Python broke it.

Wait, you can't see the CircleCI logs? I thought it was open-source. I do have to login with Github, maybe you need to do that?

To answer your question, Circle's mac builds are still on xcode:9.4.1, macOS 10.13. Which makes this test failure all the more interesting:

        # check that the expected wheels are produced
        expected_wheels = utils.expected_wheels('spam', '0.1.0')
>       assert set(actual_wheels) == set(expected_wheels)
E       AssertionError: assert {'spam-0.1.0-..._64.whl', ...} == {'spam-0.1.0-..._64.whl', ...}
E         Extra items in the left set:
E         'spam-0.1.0-cp39-cp39-macosx_10_9_universal2.whl'
E         Extra items in the right set:
E         'spam-0.1.0-cp39-cp39-macosx_10_9_x86_64.whl'
E         'spam-0.1.0-cp35-cp35m-macosx_10_9_x86_64.whl'
E         'spam-0.1.0-pp27-pypy_73-macosx_10_9_x86_64.whl'
E         Use -v to get the full diff

To me, that looks like it produced a universal2 wheel, even though it was built on xcode9 which almost certainly doesn't support Apple Silicon! Wat.

More wheel related, if you need a really new pip to use a universal2 wheel, I expect most projects will provide both for the near future - in which case, it seems much more sensible to provide a Intel and an Apple Silicon wheel instead of an Intel and a Universal2 wheel. So the question remains, can you set a target?

Did a little searching though the CPython source and found _PYTHON_HOST_PLATFORM: https://github.com/python/cpython/blob/master/Lib/distutils/util.py#L47 (thanks @Czaki for the pointer). And I remembered that we already do this on CPython 3.5, I think @mayeut figured it out. So I guess the same approach would work here.

If that works, we might as well select the universal2 Python distribution where it's available.

[Czaki]

@joerick I think I quote too long text. I comment mainly:

Can you build an Intel (or Apple Silicon) wheel from the universal Python binary?

[joerick]

Okay, this is ready for review!

The only think that will probably change is the removal of the wheel rename hack - judging by pypa/pip#9533 it looks like pip will soon support universal2 on arm64, and then we can remove that too. Thank you @henryiii for pushing that forward!

[henryiii]

Let's bump our pins and dump the hack!!!

pypa/pip#9282 (comment)

[joerick]

I can't actually run bin/update_dependencies.py locally anymore - a combination of no Docker on M1 and no CPython 3.5 on macOS 11 - so I've done it in GHA instead. I wonder if this works...

[joerick]

Ah, it does not want to make a PR on a PR. Hey ho. I'll do it on master instead and merge that in.

[henryiii]

You could let it make it on master then redirect it. But not really any different.

[henryiii]

Actually, you can just cherry pick the change here and keep going. :)

[henryiii]

Once we have access to packaging from #536 , then we should do a >= 3.9 here so we don't have to change it when we add 3.10. But this is fine until then.

[henryiii]

Could we sort and get the newest one? Not a change for this PR, but selecting the latest might be better than selecting randomly?

[joerick]

From what I can see on https://xcodereleases.com/ , each version of Xcode only ships with one macOS SDK version. I couldn't find a case where there are multiple. But I wasn't 100% sure, hence the use of lists here.

Also not sure what the selection criteria would be... would we want the newest , or the oldest? I tend to find that binaries are more often forward-compatible than backward-compatible. So then we'd want the oldest? ¯\_(ツ)_/¯

[YannickJadoul]

Wow. I haven't really been following (partially because I didn't feel like I wanted more macOS issues in my life) this intensively, but great job, everyone! :-)

[joerick]

Aaaand it's in! Thanks everyone for the input, especially @henryiii , @Czaki , @ronaldoussoren and @mayeut !

[ogrisel]

Thanks very much to everybody involved in making this happen.