Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CIBW: add support for py3.10 and Apple Silicon; use CIBWv2 #211

Merged
merged 45 commits into from
Nov 23, 2021

Conversation

MuellerSeb
Copy link
Member

@MuellerSeb MuellerSeb commented Oct 11, 2021

  • Add support for py3.10
  • Add wheels for Apple Silicon
  • split CI jobs per arch
  • CIBW config in pyproject.toml
  • update classifiers in setup.cfg

@MuellerSeb MuellerSeb added this to the 1.3.4 milestone Oct 11, 2021
@MuellerSeb MuellerSeb requested a review from LSchueler October 11, 2021 05:09
@MuellerSeb MuellerSeb self-assigned this Oct 11, 2021
@MuellerSeb
Copy link
Member Author

Guess we have to wait for py310 wheels from numpy and scipy to be released.
@LSchueler do you think we still need 32-bit wheels?

@LSchueler
Copy link
Member

I think most Raspberry Pis still run 32bit systems, if I'm not mistaken. I guess it is not very relevant, but it doesn't really hurt us at the moment to keep the 32bit wheels, so I vote for keeping them.

@MuellerSeb
Copy link
Member Author

Numpy v1.21.3 was released with prebuild wheels for python v3.10 on 64bit machines. Therefore I skipped 32bit wheels on python 3.10 for now.

Unfortunately, we have to wait until oldest-supported-numpy bumps the version for py310, since only linux wheels are provided for py310 by numpy v1.21.2: scipy/oldest-supported-numpy#26

Afterwards we have to see, if numpy adds wheels for 32bit (win+linux) for py310 and if oldest-supported-numpy will respect that.

@tupui
Copy link

tupui commented Oct 21, 2021

FYI both NumPy and SciPy will not ship 32bits wheels anymore (py3.10 and above).

@LSchueler
Copy link
Member

FYI both NumPy and SciPy will not ship 32bits wheels anymore (py3.10 and above).

Thanks for the Info!

@MuellerSeb
Copy link
Member Author

@LSchueler do you have any clue on what manylinux version to use? For now I selected manylinux2010 for all python version except Python 3.10 where I set manylinux2014.

Before we were providing manylinux1 and manylinux2010 wheels for all versions (py3.6 - py3.9).
One idea from the cibuildwheel documentation is to set manylinux1 only for py3.6 (link)

Also I didn't enable the musllinux wheels, since this is another thing I don't know to much about (and CI failed with them enabled).

So, remaining questions:

  1. Should we select manylinux1 only for py3.6?
  2. Do we need separate builds for manylinux1 and manylinux2010 for py3.6 - py3.9 like before?
  3. Do we need musllinux wheels?

@LSchueler
Copy link
Member

Good questions! And thank you for keeping everything up to date 👍

  1. I think manylinux1 has been obsolete since 2017 or something like that and the only reason to still use it that the pip version which is shipped with Python 3.6 only supports manylinux1. So I guess for the time being we should still use it for Py3.6.
  2. For the newer Python versions, I think we can safely drop the manylinux1 support. In fact, I think we can even upgrade to manylinux2014, if I'm not mistaken. You can double check it in the table here. I think with using that version, we might actually even support some non x84-systems.
  3. Did I get it right, that musllinux is mainly (only?) used for containers? I'm not sure if users who create their own container don't have the tools to let pip compile GSTools locally. On the other hand, how much work is it to support that? - I think I'm slightly leaning towards not supporting musllinux, but I really don't have a very strong opinion on that topic.

Now, the GSTools-Core is used, if the package can be imported, but it
can also be switched off, by setting the global var.
`gstools.config.USE_RUST=False` during the runtime.
This makes the Cython code compatible to GSTools-Core v0.1.2 again.
Add optional dependency GSTools-Core
@MuellerSeb
Copy link
Member Author

1. I think `manylinux1` has been obsolete since 2017 or something like that and the only reason to still use it that the `pip` version which is shipped with Python 3.6 only supports `manylinux1`. So I guess for the time being we should still use it for Py3.6.

Will do so.

2. For the newer Python versions, I think we can safely drop the `manylinux1` support. In fact, I think we can even upgrade to `manylinux2014`, if I'm not mistaken. You can double check it in the table [here](https://github.com/pypa/manylinux). I think with using that version, we might actually even support some non x84-systems.

Ok, so to keep brainwork low on our side, I would then just use the defaults of CIBuildwheel, which is manylinux2010 ATM.

3. Did I get it right, that `musllinux` is mainly (only?) used for containers? I'm not sure if users who create their own container don't have the tools to let pip compile GSTools locally. On the other hand, how much work is it to support that? - I think I'm slightly leaning towards not supporting `musllinux`, but I really don't have a very strong opinion on that topic.

It didn't work, since LAPACK and BLAS were missing and the scipy installation was failing. I think it would need some effort to get this working and following your arguments: people building a container should be able to build it from source.

Afterwards we should be fine.

Copy link
Member

@LSchueler LSchueler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, Py3.10 support!
I think a release of v1.3.4 makes sense now (the doc. for how to use GSTools-Core doesn't make a lot of sense without the release and now the support for py3.10). Or would you rather wait for the paper to get published?

@MuellerSeb
Copy link
Member Author

@LSchueler when the CI succeeds, I would merge this. A 1.3.4 release would be just fine afterwards. Maybe we could wait for the missing wheels of GSTools-Core: GeoStat-Framework/GSTools-Core#15

@MuellerSeb MuellerSeb changed the title CIBW: add support for py3.10; use CIBWv2 CIBW: add support for py3.10 and Apple Silicon; use CIBWv2 Nov 17, 2021
@MuellerSeb
Copy link
Member Author

I did a rebase to update the changelog. Guess this messed up the commits. Should be fine afterwards.
I now also added:

  • wheels for Apple Silicon (arm64 and universal2)
  • separate CI jobs per arch
  • updated classifiers in setup.cfg

Copy link
Member

@LSchueler LSchueler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is really messed up! :-D
But yeah, I'm fine with it 👍
LGTM!

@MuellerSeb MuellerSeb merged commit d8941c2 into main Nov 23, 2021
@MuellerSeb MuellerSeb deleted the CIBW_update branch November 23, 2021 10:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants