Consider uploading wheels to pypi to avoid needing compilation when installing

[cgc]

I use a library (nipype) that uses traits. I often package my code that uses nipype/traits with docker, but unfortunately, traits has to build from source, so I have to install python-dev in my ubuntu docker images, which I'm hoping to stop doing.

Have you given any consideration to building and uploading wheels for common platforms when uploading to pypi? I understand this may be a significant departure from the current upload procedure.

[kaczmarj]

I would also appreciate python wheels for this project. Wheels would make it possible to install traits in a fresh environment (e.g., a Docker container).

For what it's worth, python wheels can be created by running the following line in the project root directory.

python setup.py bdist_wheel

The wheels can be uploaded to pypi with twine.

The tricky thing is that a wheel would have to be created for each platform (docs).

@cgc - You can also install traits from conda-forge. conda distributes pre-built packages. https://anaconda.org/conda-forge/traits

[ericpre]

Is there any hope to have wheels build on travis (and appveyor for example) and uploaded on pypi? @kaczmarj explained nicely how it can be done easily.

[kaczmarj]

The python documentation for packaging C extensions has been updated: https://packaging.python.org/guides/packaging-binary-extensions/#building-binary-extensions

For Linux, one can build the project inside a Docker image (available at https://github.com/pypa/manylinux)

For Windows, install Visual Studio 2015 Community Edition or later, and build the project.

For macOS:

Binary compatibility on macOS is determined by the target minimum deployment system, e.g. 10.9, which is often specified with the MACOSX_DEPLOYMENT_TARGET environmental variable when building binaries on macOS. When building with setuptools / distutils, the deployment target is specified with the flag --plat-name, e.g. macosx-10.9-x86_64. For common deployment targets for macOS Python distributions, see the MacPython Spinning Wheels wiki.

[mdickinson]

It would definitely be good to have wheels available, but I'm afraid actually doing the work for that is unlikely to be a priority for anyone at Enthought any time soon. Our own projects don't need this (we have our own deployment solution), so it's difficult to justify anyone spending their work hours on this.

That leaves open the possibility of someone doing this in their personal time, of course. (And I know that in my own personal time this isn't going to be high on my priority list.)

[mdickinson]

FWIW, it looks like we'd want at a minimum 12 wheels (4 Python versions times three platforms: macOS / 64-bit manylinux / 64-bit Windows). It's probably not worth the effort of building wheels for 32-bit platforms unless/until someone indicates that they need those.

[cboulay]

In case this helps, here is an AppVeyor script that uses twine to build python wheels for different platforms, then, if the commit was a tagged release, upload them to pypi:
https://github.com/labstreaminglayer/liblsl-Python/blob/master/appveyor.yml

However, all this does is download precompiled libraries as the library is a ctypes library. The difficult compilation from source is done on CI servers using a mixture of services (at the time there wasn't a single service that supported all the platforms we support so we used AppVeyor, Travis, Azure, Github workflows). For the manylinux platforms we used azure: https://github.com/sccn/liblsl/blob/master/azure-pipelines.yml

GitHub workflows and Azure Pipelines have matured quite a bit since then, so you could probably roll a single CI workflow to build from source and upload to pypi for all your target platforms.

[mdickinson]

GitHub workflows and Azure Pipelines have matured quite a bit since then [...]

Yes, it's now looking as though it ought to be possible to do this using just GitHub Actions. There's some related documentation here: https://packaging.python.org/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows

I'm considering moving our CI to GitHub Actions anyway: see #1287 for a POC.

[mdickinson]

I spent last weekend working through the details of automated wheel building for another project: see https://github.com/enthought/ibm2ieee/blob/master/.github/workflows/upload-to-pypi.yml; the details in #1291 helped - thank you @ericpre. I think we should now be in a position to implement this for Traits.

[mdickinson]

The one change I'd make from the ibm2ieee version would be to put the sdist build and upload into a separate job from the wheel build and upload.

[mdickinson]

One minor awkwardness to note: ideally for macOS we'd build wheels that are compatible with older versions of macOS, but the Python builds supplied by the setup-python action were built with MACOSX_DEPLOYMENT_TARGET=10.14, so extensions built will not be compatible with macOS 10.13 and earlier. I think we can live with that.

I also have no idea how arm64 builds should work. (Related: actions/setup-python#108)

Related: actions/setup-python#26

[mdickinson]

Re-opening this. It's solved in principle (the workflow to build and upload wheels exists), but not yet solved in practice (there are no wheels on PyPI). I'll close once those wheels exist and are known good.

[corranwebster]

As of Traits 6.2.0 wheels are now uploaded on PyPI.

[ericpre]

Great, thanks, just to confirm that the wheels are working well!

[mdickinson]

Thanks, @ericpre; good to know. Out of interest, which platform are you using the wheels on?

[ericpre]

I have tested it on windows (amd64) and linux (x86_64).