Support for easier parallel builds for different architectures

[filips123]

Currently, the only parallel thing that happens is building for different operating systems. However, all builds in that specific OS will still be sequential which means bigger projects will take very long time to compile. However, most CI systems support parallel builds. If you buld wheels for 32-bit and 64-bit in parallel, you will get much faster builds.

You can already do this if you use CIBW_BUILD and separatly define once 32-bit and once 64-bit builds for each system (except macOS which only has 64-bit). However, problem is that Windows and Linux use different architecture tags: Windows uses amd64 and win32, while (many)Linux uses x86_64 and i686 (and other architectures as well). This means it is harder to define them in "compact" and easy way.

For example, if architecture tags would be the same, you could use something like this (in GitHub Actions):

runs-on: ${{ matrix.os }}
strategy:
  matrix:
    architecture: [x64, x86]
    os: [ubuntu-latest, macos-latest, windows-latest]
    exclude:
      - architecture: x86
        os: macos-latest
env:
  CIBW_BUILD_ARCH: ${{ matrix.architecture }}

But if it is not, you will have to manually define more jobs with separate matrices for separate tags.

Maybe there is also some better way, but it would be nice if this is also specified in documentation and examples.

[Czaki]

There are multiple options.
simplest is to use CIBW_SKIP with multiple entries (CIBW_SKIP: "*win32 *i686" for x64)
So it will looks like

runs-on: ${{ matrix.os }}
strategy:
  matrix:
    architecture: [x64, x86]
    os: [ubuntu-latest, macos-latest, windows-latest]
    exclude:
      - architecture: x86
        os: macos-latest
    include:
      - architecture: x86
        CIBW_SKIP: "*64"
      - architecture: x64
        CIBW_SKIP: "*win32 *i686"

[filips123]

Unfortunately this does not work because it seems you can't conditionally define environment variables (so that their values are different than values in matrix, but they still conditionally depend on values in matrix) in easy way (you could do this with another build/job step but that can be too much work). I came out with this which enables you to define different environment variable values for different matrix jobs:

name: Python bindings
on: [push, pull_request]

env:
 CIBW_SKIP: cp27-* pp*

jobs:
  build:
    name: Building and testing on ${{ matrix.config.os }} ${{ matrix.config.architecture }}
    runs-on: ${{ matrix.config.os }}

    strategy:
      fail-fast: false
      matrix:
        config:
        - {os: ubuntu-latest, architecture: "*x86_64" }
        - {os: ubuntu-latest, architecture: "*i686" }
        - {os: windows-latest, architecture: "*amd64" }
        - {os: windows-latest, architecture: "*win32" }
        - {os: macos-latest, architecture: "*x86_64" }

    env:
      CIBW_BUILD: ${{ matrix.config.architecture }}

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Set up Python ${{ matrix.python_version }}
      uses: actions/setup-python@v1
      with:
        python-version: 3.7

    - name: Install cibuildwheel
      run: python -m pip install cibuildwheel

    - name: Build and test wheels
      run: python -m cibuildwheel

    - name: Upload artifacts to storage
      if: success() || failure()
      uses: actions/upload-artifact@v2
      with:
        path: ./wheelhouse/*.whl

It seems to work, although it may not be the best or easiest solution.

[joerick]

That looks good to me @filips123. What kind of concurrency limits do Github apply in this scenario? Is there a limit to the number of active jobs?

[YannickJadoul]

I'm a little hesitant to add yet another way of selecting/unselecting builds. How are all these going to interact?
Also, I'm not entirely convinced we should tailor cibuildwheel to closely to a single CI platforms's conventions of architecture identifiers?

[filips123]

What kind of concurrency limits do Github apply in this scenario? Is there a limit to the number of active jobs?

20 concurrent jobs for whole account for free plan, 40 for pro plan, with maximum of 5 concurrent macOS jobs.

See this for more details.

I'm a little hesitant to add yet another way of selecting/unselecting builds. How are all these going to interact?

Yes, I agree. I seems that my solution with manually specifying existing platform tags works, and it isn't that much work. Other CI services also probably support similar ways of specifying jobs.

Maybe it would be just nice to have some examples or documentation about this.

[YannickJadoul]

Maybe it would be just nice to have some examples or documentation about this.

Silly idea, maybe, probably. But how hard is it to create a GitHub action? If it's possible to somehow put the responsibility of splitting up the cibuildwheel job into a GitHub action (e.g., automatically setting CIBW_BUILD), this could be solved outside of cibuildwheel?

[henryiii]

Does #469 help here?

[henryiii]

I think #482 and the followup #535, combined with the current actions support, likely covers this pretty well. We could add an "arch" setting to the action, and allow --archs="" to be treated like None, but I don't think this is really better than the environment variable? It's only "special" because it's allowed as an option to cibuildwheel, so it might also be a useful option to add to the action. We don't allow "platform", but that would be silly. I'll probably make a quick PR to suggest it, so see if we want it, but regardless I think this is closable.

Oh, actually, what if we allow activate QUEMU on the action if this is passed, though? That would be useful! Though maybe tricky, we can't call actions from actions. I think there are examples. I'll check.