Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cross-build wheel jobs for ppc64le and s390x #7549

Merged

Conversation

mtreinish
Copy link
Member

Summary

This commit adds a new CI job at release time to build precompiled
binaries for s390x and ppc64le linux platforms. These platforms do not
have precompiled wheels for any upstream dependencies (except for
retworkx and tweedledum, which both publish binaries for both platforms).
Since the only publically available CI service that has native s390x and
ppc64le environments available is travis which doesn't provide
sufficient quota to open source projects anymore to make it usable we
have to rely on either cross compilation or emulation. Cross compiling
with python wheels while possible is quite tricky to setup and configure
in practice so emulation is used here. This is a much simpler path
configure, especially because cibuildwheel has support for emulating
non-x86 architectures via QEMU to build wheels. While QEMU emulation of
other architectures is exceedingly slow, we have a 6 hour job time limit
with github actions which should hopefully be sufficient to compile the
binary and build wheels for all our supported python versions.

This was previously attempted in #5844 but we hit a timout issue
reliably every time trying to build scipy from source. Compiling scipy
is slow on a native system and it's an order of magnitude slower under
qemu emulation. In #5844 we exceeded the 6 hour job time limit just
compiling scipy once for testing. To avoid this issue (since scipy
doesn't publish s390x or ppc64le wheels) this commit skips the test
runs on these jobs. This means we're solely building the wheels and not
testing if they're valid. This also means installing scipy and other
dependencies is still an exercise for the s390x and ppc64le users. But,
if this works it will mean that at least for Qiskit there will not be a
need to compile anything from source.

Details and comments

This commit adds a new CI job at release time to build precompiled
binaries for s390x and ppc64le linux platforms.  These platforms do not
have precompiled wheels for any upstream dependencies (except for
retworkx and tweedledum, which both publish binaries for both platforms).
Since the only publically available CI service that has native s390x and
ppc64le environments available is travis which doesn't provide
sufficient quota to open source projects anymore to make it usable we
have to rely on either cross compilation or emulation. Cross compiling
with python wheels while possible is quite tricky to setup and configure
in practice so emulation is used here. This is a much simpler path
configure, especially because cibuildwheel has support for emulating
non-x86 architectures via QEMU to build wheels. While QEMU emulation of
other architectures is exceedingly slow, we have a 6 hour job time limit
with github actions which should hopefully be sufficient to compile the
binary and build wheels for all our supported python versions.

This was previously attempted in Qiskit#5844 but we hit a timout issue
reliably every time trying to build scipy from source. Compiling scipy
is slow on a native system and it's an order of magnitude slower under
qemu emulation. In Qiskit#5844 we exceeded the 6 hour job time limit just
compiling scipy once for testing. To avoid this issue (since scipy
doesn't publish s390x or ppc64le wheels) this commit skips the test
runs on these jobs. This means we're solely building the wheels and not
testing if they're valid. This also means installing scipy and other
dependencies is still an exercise for the s390x and ppc64le users. But,
if this works it will mean that at least for Qiskit there will not be a
need to compile anything from source.
@mtreinish mtreinish requested a review from a team as a code owner January 20, 2022 15:16
@mtreinish mtreinish added the on hold Can not fix yet label Jan 20, 2022
@mtreinish mtreinish added the type: qa Issues and PRs that relate to testing and code quality label Jan 20, 2022
This commit should not be merged without being reverted first. It is
changing the CI trigger to run on open pull requests and removing the
twine upload step to trigger the wheel builds for testing and
iteratively fixing issues. Once the builds work this will be reverted
and the PR is ready for final review/merging.
@coveralls
Copy link

coveralls commented Jan 20, 2022

Pull Request Test Coverage Report for Build 1728983295

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 83.15%

Totals Coverage Status
Change from base Build 1726040390: 0.0%
Covered Lines: 52001
Relevant Lines: 62539

💛 - Coveralls

Testing shows this works well and we can build and publish ppc64le and
s390x wheels on release.

This reverts commit f06c0b6 and opens
up the PR for review.
@mtreinish mtreinish removed the on hold Can not fix yet label Jan 20, 2022
@mtreinish
Copy link
Member Author

Ci worked, you can see the logs (and download the wheels) from here: https://github.com/Qiskit/qiskit-terra/actions/runs/1724756938

I'll spin up an s390x vm in a bit to test the wheel works

Copy link
Member

@jakelishman jakelishman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In principle this looks good to me, but I don't really have the experience with virtualisation to meaningfully test the output wheels (nor the right software on my laptop, as far as I know!).

Do we want a release note to talk about the new wheels?

@mtreinish
Copy link
Member Author

In principle this looks good to me, but I don't really have the experience with virtualisation to meaningfully test the output wheels (nor the right software on my laptop, as far as I know!).

You could install QEMU locally for emulation like we do in CI to build these but I don't really recommend this as it's super slow. I normally test s390x via the LinuxONE community cloud which let you provision s390x VMs for open source work:

https://developer.ibm.com/gettingstarted/ibm-linuxone/

(my account was stale so I'm waiting on a new one to be provisioned now). For ppc64le I don't have a good test mechanism short of knowing someone with a server.

Do we want a release note to talk about the new wheels?

Yeah a release note is a good call, I'll add one in a bit.

mtreinish added a commit to mtreinish/qiskit-core that referenced this pull request Jan 21, 2022
In Qiskit#7102 we added support for Python 3.10 to Qiskit. However, numpy and
scipy stopped publishing wheels for 32bit platforms in Python 3.10. This
means that to test Python 3.10 32bit wheels we will have to compile
numpy and scipy from source (which for scipy is prohibitively slow).
In response this commit updates our cibuildwheel configuration to skip
tests on 32bit Python 3.10 wheels since we can't reliably test the built
wheels in CI and it would require users to build them from source.
This is the same approach taken in PR Qiskit#7549 for ppc64le and s390x. For
32 bit platform users on python 3.10 this means they'll need to locally
build numpy, scipy, and potentially other depencies from source to use
Qiskit Terra with Python 3.10, but for the Qiskit components at least
precompiled binaries will be available.
@jakelishman jakelishman added the Changelog: New Feature Include in the "Added" section of the changelog label Jan 21, 2022
@jakelishman
Copy link
Member

Approved, provided you're happy with your latest VM tests. It might not worth backporting this to 0.19.2, unless you reckon it'd be good to do a test release with the new platforms before the next minor.

mtreinish added a commit to mtreinish/qiskit that referenced this pull request Jan 21, 2022
The supported platform list in the install documentation wasn't quite up
to date. The supported platforms are also a bit more nuanced than that
list made it appear. This commit expands the platform support section
and documents the different levels of support for platforms and the
details about each level. This should hopefully make it clearer to users
which platforms are expected to work and what guarantees are made for
each platform.

This is related to Qiskit/qiskit#7553, Qiskit/qiskit#7549,
and Qiskit/qiskit-aer#1362 which are expanding (or just changing the
level of support) the supported platforms but each have a different set
of tradeoffs that we should be documenting.
@mtreinish
Copy link
Member Author

I've tested this on s390x (with the python 3.8 wheel) and it works fine (after compiling numpy and scipy from source):

Python 3.8.10 (default, Nov 26 2021, 20:14:08) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.processor()
's390x'
>>> import qiskit
>>> qiskit.__version__
'0.20.0'
>>> qc = qiskit.QuantumCircuit(2)
>>> qc.h(0)
<qiskit.circuit.instructionset.InstructionSet object at 0x3ff7e6bc080>
>>> qc.cx(0, 1)
<qiskit.circuit.instructionset.InstructionSet object at 0x3ff7e6bc340>
>>> qc.measure_all()
>>> from qiskit.test.mock import FakeRome
>>> print(qiskit.execute(qc, FakeRome()).result().get_counts())
/home/linux1/test_wheels/lib/python3.8/site-packages/qiskit/test/mock/fake_backend.py:156: RuntimeWarning: Aer not found using BasicAer and no noise
  warnings.warn("Aer not found using BasicAer and no noise", RuntimeWarning)
{'11': 521, '00': 503}

@mtreinish
Copy link
Member Author

mtreinish commented Jan 21, 2022

Approved, provided you're happy with your latest VM tests. It might not worth backporting this to 0.19.2, unless you reckon it'd be good to do a test release with the new platforms before the next minor.

Yeah, I think we can just hold this until 0.20.0 (this will also give us time to do a similar PR on aer). There's no pressing need here. It's not like either of these platforms have a lot of users. Looking at the CPU stats from PyPI over the last 30 days:

| cpu      | download_count |
| -------- | -------------- |
| x86_64   |         52,446 |
| None     |         25,158 |
| AMD64    |          9,037 |
| arm64    |            465 |
| aarch64  |            198 |
| armv7l   |            138 |
| i686     |            136 |
| s390x    |             58 |
| iPad13,4 |             24 |
| iPad8,5  |             20 |
| Total    |         87,680 |

s390x clients are only 0.06% of our downloads and 0 ppc64le downloads.

jakelishman added a commit to Qiskit/qiskit-metapackage that referenced this pull request Jan 21, 2022
* Expand platform support section to install doc

The supported platform list in the install documentation wasn't quite up
to date. The supported platforms are also a bit more nuanced than that
list made it appear. This commit expands the platform support section
and documents the different levels of support for platforms and the
details about each level. This should hopefully make it clearer to users
which platforms are expected to work and what guarantees are made for
each platform.

This is related to Qiskit/qiskit#7553, Qiskit/qiskit#7549,
and Qiskit/qiskit-aer#1362 which are expanding (or just changing the
level of support) the supported platforms but each have a different set
of tradeoffs that we should be documenting.

* Apply suggestions from code review

Co-authored-by: Jake Lishman <jake@binhbar.com>

Co-authored-by: Jake Lishman <jake@binhbar.com>
@jakelishman jakelishman added this to the 0.20 milestone Jan 21, 2022
@mergify mergify bot merged commit b5f87a6 into Qiskit:main Jan 21, 2022
@mtreinish mtreinish deleted the attempt-cross-compilation-ibm-platforms branch January 21, 2022 17:55
mergify bot pushed a commit that referenced this pull request Jan 21, 2022
* Drop support for 32bit py310 wheel builds

In #7102 we added support for Python 3.10 to Qiskit. However, numpy and
scipy stopped publishing wheels for 32bit platforms in Python 3.10. This
means that to test Python 3.10 32bit wheels we will have to compile
numpy and scipy from source (which for scipy is prohibitively slow).
In response this commit updates our cibuildwheel configuration to skip
tests on 32bit Python 3.10 wheels since we can't reliably test the built
wheels in CI and it would require users to build them from source.
This is the same approach taken in PR #7549 for ppc64le and s390x. For
32 bit platform users on python 3.10 this means they'll need to locally
build numpy, scipy, and potentially other depencies from source to use
Qiskit Terra with Python 3.10, but for the Qiskit components at least
precompiled binaries will be available.

* Update release note
@hhorii
Copy link
Contributor

hhorii commented Jan 24, 2022

I've tested this on ppc64le (with python 3.7, 3.8, and 3.9) and they worked fine.

$ python
Python 3.7.11 (default, Jul 27 2021, 15:02:35) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.processor()
'ppc64le'
>>> import qiskit
>>> qiskit.__version__
'0.20.0'
>>> qc = qiskit.QuantumCircuit(2)
>>> qc.h(0)
<qiskit.circuit.instructionset.InstructionSet object at 0x7dd35f50caa0>
>>> qc.cx(0, 1)
<qiskit.circuit.instructionset.InstructionSet object at 0x7dd35f50cdc0>
>>> qc.measure_all()
>>> from qiskit.test.mock import FakeRome
>>> print(qiskit.execute(qc, FakeRome()).result().get_counts())
/home/hhorii/anaconda3/envs/qiskit202201_3.7/lib/python3.7/site-packages/qiskit/test/mock/fake_backend.py:156: RuntimeWarning: Aer not found using BasicAer and no noise
  warnings.warn("Aer not found using BasicAer and no noise", RuntimeWarning)
{'00': 502, '11': 522}

jakelishman pushed a commit to jakelishman/qiskit-terra that referenced this pull request Jan 24, 2022
* Drop support for 32bit py310 wheel builds

In Qiskit#7102 we added support for Python 3.10 to Qiskit. However, numpy and
scipy stopped publishing wheels for 32bit platforms in Python 3.10. This
means that to test Python 3.10 32bit wheels we will have to compile
numpy and scipy from source (which for scipy is prohibitively slow).
In response this commit updates our cibuildwheel configuration to skip
tests on 32bit Python 3.10 wheels since we can't reliably test the built
wheels in CI and it would require users to build them from source.
This is the same approach taken in PR Qiskit#7549 for ppc64le and s390x. For
32 bit platform users on python 3.10 this means they'll need to locally
build numpy, scipy, and potentially other depencies from source to use
Qiskit Terra with Python 3.10, but for the Qiskit components at least
precompiled binaries will be available.

* Update release note

(cherry picked from commit 371a2bc)
jakelishman pushed a commit that referenced this pull request Jan 25, 2022
* Drop support for 32bit py310 wheel builds

In #7102 we added support for Python 3.10 to Qiskit. However, numpy and
scipy stopped publishing wheels for 32bit platforms in Python 3.10. This
means that to test Python 3.10 32bit wheels we will have to compile
numpy and scipy from source (which for scipy is prohibitively slow).
In response this commit updates our cibuildwheel configuration to skip
tests on 32bit Python 3.10 wheels since we can't reliably test the built
wheels in CI and it would require users to build them from source.
This is the same approach taken in PR #7549 for ppc64le and s390x. For
32 bit platform users on python 3.10 this means they'll need to locally
build numpy, scipy, and potentially other depencies from source to use
Qiskit Terra with Python 3.10, but for the Qiskit components at least
precompiled binaries will be available.

* Update release note

(cherry picked from commit 371a2bc)
mergify bot added a commit that referenced this pull request Jan 26, 2022
* Add support for Python 3.10 (backport #7102)

Merge notes: for the backport, we must ensure that Python 3.6 is still
supported.

* Add support for Python 3.10

Python 3.10.0 was released on 10-04-2021, this commit marks the support
of Python 3.10 in qiskit-terra. It adds the supported python version in
the package metadata and updates the CI configuration to run test jobs
on Python 3.10 and build Python 3.10 wheels.

* Fix typo

* Update default envlist in tox.ini to include 3.10

* Bump cibuildwheel to the latest version

This also takes the opportunity to deduplicate the cibuildwheel
configuration using the pyproject.toml support in newer versions of
cibuildwheel. The common options for all builds are put there and per
build overrides (which are only for cross compiling arm wheels) are left
as environment variables in the CI configuration.

* Add missing cibuildwheel config to pyproject.toml

* Ignore internal deprecation warning emitted by jupyter in ci

* Fix black

(cherry picked from commit 9a743fb)

Co-authored-by: Jake Lishman <jake.lishman@ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Skip musl builds of linux

Co-authored-by: Matthew Treinish <mtreinish@kortar.org>

* Update Windows wheel build VM

The `vs2017-windows2016` is deprecated, and due to be removed in a few
months; it is currently in the brownout period.  `windows-latest` now
contains the VS build tools we were using, so should be sufficient.

* Drop support for 32bit py310 wheel builds (backport #7553)

* Drop support for 32bit py310 wheel builds

In #7102 we added support for Python 3.10 to Qiskit. However, numpy and
scipy stopped publishing wheels for 32bit platforms in Python 3.10. This
means that to test Python 3.10 32bit wheels we will have to compile
numpy and scipy from source (which for scipy is prohibitively slow).
In response this commit updates our cibuildwheel configuration to skip
tests on 32bit Python 3.10 wheels since we can't reliably test the built
wheels in CI and it would require users to build them from source.
This is the same approach taken in PR #7549 for ppc64le and s390x. For
32 bit platform users on python 3.10 this means they'll need to locally
build numpy, scipy, and potentially other depencies from source to use
Qiskit Terra with Python 3.10, but for the Qiskit components at least
precompiled binaries will be available.

* Update release note

(cherry picked from commit 371a2bc)

* Use manylinux2014 on Python 3.10+

Numpy and Scipy are moving to drop manylinux2010 wheels on newer
platforms, which gives us some cover to do the same.  For the 0.19
series, though, we need to maintain the same manylinux compatibility we
had before.  To test, we need to ensure that we pull Numpy and Scipy in
binary form only (forcing pip to install slightly older versions like
1.21 instead of 1.22), rather than attempting to build Numpy from source
as part of our testing process.

Co-authored-by: Matthew Treinish <mtreinish@kortar.org>
Co-authored-by: Jake Lishman <jake.lishman@ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
jakelishman added a commit to jakelishman/qiskit-terra that referenced this pull request Aug 11, 2023
…kage#1402)

* Expand platform support section to install doc

The supported platform list in the install documentation wasn't quite up
to date. The supported platforms are also a bit more nuanced than that
list made it appear. This commit expands the platform support section
and documents the different levels of support for platforms and the
details about each level. This should hopefully make it clearer to users
which platforms are expected to work and what guarantees are made for
each platform.

This is related to Qiskit#7553, Qiskit#7549,
and Qiskit/qiskit-aer#1362 which are expanding (or just changing the
level of support) the supported platforms but each have a different set
of tradeoffs that we should be documenting.

* Apply suggestions from code review

Co-authored-by: Jake Lishman <jake@binhbar.com>

Co-authored-by: Jake Lishman <jake@binhbar.com>
SamD-1998 pushed a commit to SamD-1998/qiskit-terra that referenced this pull request Sep 7, 2023
…kage#1402)

* Expand platform support section to install doc

The supported platform list in the install documentation wasn't quite up
to date. The supported platforms are also a bit more nuanced than that
list made it appear. This commit expands the platform support section
and documents the different levels of support for platforms and the
details about each level. This should hopefully make it clearer to users
which platforms are expected to work and what guarantees are made for
each platform.

This is related to Qiskit#7553, Qiskit#7549,
and Qiskit/qiskit-aer#1362 which are expanding (or just changing the
level of support) the supported platforms but each have a different set
of tradeoffs that we should be documenting.

* Apply suggestions from code review

Co-authored-by: Jake Lishman <jake@binhbar.com>

Co-authored-by: Jake Lishman <jake@binhbar.com>
Eric-Arellano pushed a commit to Qiskit/documentation that referenced this pull request Oct 12, 2023
…kage#1402)

* Expand platform support section to install doc

The supported platform list in the install documentation wasn't quite up
to date. The supported platforms are also a bit more nuanced than that
list made it appear. This commit expands the platform support section
and documents the different levels of support for platforms and the
details about each level. This should hopefully make it clearer to users
which platforms are expected to work and what guarantees are made for
each platform.

This is related to Qiskit/qiskit#7553, Qiskit/qiskit#7549,
and Qiskit/qiskit-aer#1362 which are expanding (or just changing the
level of support) the supported platforms but each have a different set
of tradeoffs that we should be documenting.

* Apply suggestions from code review

Co-authored-by: Jake Lishman <jake@binhbar.com>

Co-authored-by: Jake Lishman <jake@binhbar.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Changelog: New Feature Include in the "Added" section of the changelog type: qa Issues and PRs that relate to testing and code quality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants