Build and distribute pyosmium wheel

[daniel-j-h]

Do you think it makes sense to build and distribute a Python Wheel for PyOsmium? At the moment users need to install build-essentials, all libosmium native deps, and boost-python-dev.

Would it make sense to build a wheel and ship it, so that users can simply

pip install osmium

and don't have to worry about dependency management?

(FYI: this is food for thought, not urgent or blocking - I noticed this while bundling a pyosmium program with Docker and was a bit surprised how much I have to pull in to get pyosmium to compile)

Here are some notes with refs to get started:

• Build and Distribute Python Wheel scanse/sweep-sdk#54
• Add setuputils script for py bindings scanse/sweep-sdk#51

and it's a bit tricky with native deps as usual :)

[lonvia]

This would absolutely make sense but I don't currently have the resources to build and test this sufficiently. Leaving this issue open anyway, just in case somebody wants to pick it up.

[wiktorn]

I'm giving this a look. After some preliminary testing it looks like manylinux / PEP513 is a way to go. Just simple move from one Ubuntu release to the other breaks standard wheel because of difference in version of boost library.

The only drawback of using manylinux is that this is very conservative environment with gcc 4.8, boost 1.41 which are quite old.

I still haven't tested how standard wheels behave on Windows and macOS, but looking at this ticket I fear that it might be not that easy with Windows.

As for testing - I plan to use Travis /AppVeyor to build a wheel, and then run test suite in separate virtualenv, after, if possible, deinstallation of system-wide build packages (boost, boost-python etc.).

[joto]

PyOsmium uses libosmium which doesn't work with boost 1.41 and I am not sure it all works with GCC 4.8.

[wiktorn]

Sorry, boost 1.48. At least it didn't complain during compilation, though I have problems during linking phase. Gcc 4.8 is used by travis builds so I assume, that’s what it works. Question is, how long we want to stick to it.

OTOH I could just run auditwheel on Ubuntu trusty/xenial and see on how many platforms it will work

[mehdisadeghi]

I can test it on Mac provided the instructions.

[bnaul]

@wiktorn I'd suggest taking a look at https://github.com/matthew-brett/multibuild, I've used these tools to build OS X + manylinux wheels for a couple of packages w/ minimal effort.

You might also take a look at the ray project which builds manylinux wheels that use boost.

[wiktorn]

@bnaul Thank's a lot, that could help a lot.

Just as I got manylinux wheels up & running on Debian / Ubuntu / Centos / Arch.

[wiktorn]

Looks like I've got it working on Linux & macOS with Travis.

On Linux I build manylinux binaries using Boost 1.67 compiled from source (as CentOS 5 has outdated Boost). Using Docker build is tested on:

Distribution	Python 2	Python 3
Debian (stretch / latest)	2.7	3.5
Ubuntu (bionic / latest)	2.7	3.6
Centos (centos7 / latest)	2.7	3.4
ArchLinux (latest)	2.7	3.6

It should be fairly easy to add additional distributions to test. Though they are only x64.

On macOS builds use Boost provided by HomeBrew but I'm afraid that wheel will be pinned to specific Boost version and will fail to load if there is no specific version of Boost available on system. I'm not sure, if it's an improvement.

@lonvia Travis CI doesn't store any build artifacts. I tried uploads to AWS S3 and as a GitHub release. Both has some drawbacks - AWS needs an account there, GitHub releases are created per job, so for one build - you get 6 releases. You can always use your own build infrastructure. Which way you plan to go with this?

Next I plan to check Windows builds.

Should I test any other distributions? Anything else that should be done?

I've made also a separate repo using Matthew Brett Multibuild here:
https://github.com/wiktorn/pyosmium-wheel-build

For now it still fails on macOS because of some Heisenbug I didn't yet manage to track. @lonvia which way do you prefer?

[wiktorn]

Waiting for @lonvia to finish replacing boost-python with pybind11 and then this will be far easier to build and distribute

[lonvia]

@wiktorn I was looking a bit through the different build options. Building locally is unfortunately out of the question because I cannot build for MacOS. This only leaves travis. My preference would be to have a separate build repo as you already started with pyosmium-wheel-build and then simply using Github Releases for the artefacts. If things work out find we could also consider pushing to PyPI directly later.

[lonvia]

@wiktorn What is the status on this? You said that you wanted to do some more testing. Given the rather large changes with pybind11 and the binary wheels, I'd like to roll out an exceptional pre-release, so that we can get a bit more feedback before the next full release. Also would help to test the whole workflow of wheel creation.

[wiktorn]

Repository is moved to https://github.com/osmcode/pyosmium-wheel-build and you should have commit bit there (at least I gave it to you just before transferring it to osmcode organization).

You can try now to update pyosmium, libosmium and other dependencies to desired versions and run the Travis build. As far as I can guess, Travis build still uses my API key to upload artefacts to GitHub (pyosmium-wheel-build) so it should work. Once you're satisfied with the build artefacts you can publish them wherever you want.

Latest Travis failure looks like a temporary problem (performance / network access).

And as for testing - I can help with testing pre-release version.

[wiktorn]

I've did some tests installing binary wheel from this release:
https://github.com/osmcode/pyosmium-wheel-build/releases/tag/untagged-2289a1529af3fba1b920

Success:

# Python2
docker run -it --rm -v /home/w/Downloads:/io ubuntu:latest /bin/bash -c "apt update && apt install -y python-pip && pip install nose mock && pip install --no-index osmium --find-links /io && cd /io/test && python -m nose && python -V"
docker run -it --rm -v /home/w/Downloads:/io debian:latest /bin/bash -c "apt update && apt install -y python-pip && pip install nose mock && pip install --no-index osmium --find-links /io && cd /io/test && python -m nose && python -V"
docker run -it --rm -v /home/w/Downloads:/io fedora:latest /bin/bash -c "yum install -y python-pip && pip install nose mock && pip install --no-index osmium --find-links /io && cd /io/test && python -m nose && python -V"
docker run -it --rm -v /home/w/Downloads:/io base/archlinux:latest /bin/bash -c "pacman -Sy --noconfirm python2-pip && pip2 install nose mock && pip2 install --no-index osmium --find-links /io && cd /io/test && python2 -m nose && python2 -V"


# Python3
docker run -it --rm -v /home/w/Downloads:/io ubuntu:latest /bin/bash -c "apt update && apt install -y python3-pip && pip3 install nose mock && pip3 install --no-index osmium --find-links /io && cd /io/test && python3 -m nose"
docker run -it --rm -v /home/w/Downloads:/io debian:latest /bin/bash -c "apt update && apt install -y python3-pip && pip3 install nose mock && pip3 install --no-index osmium --find-links /io && cd /io/test && python3 -m nose"
docker run -it --rm -v /home/w/Downloads:/io fedora:latest /bin/bash -c "yum install -y python3-pip && pip3 install nose mock && pip3 install --no-index osmium --find-links /io && cd /io/test && python3 -m nose && python3 -V"
docker run -it --rm -v /home/w/Downloads:/io base/archlinux:latest /bin/bash -c "pacman -Sy --noconfirm python-pip && pip install nose mock && pip install --no-index osmium --find-links /io && cd /io/test && python -m nose && python -V"

Failure:

# Python2:
docker run -it --rm -v /home/w/Downloads:/io alpine:latest /bin/ash -c "apk update && apk add py2-pip && pip install nose mock && pip install --no-index osmium --find-links /io && cd /io/test && python -m nose && python -V"


# Python3
docker run -it --rm -v /home/w/Downloads:/io alpine:latest /bin/ash -c "apk update && apk add py3-pip && pip3 install nose mock && pip3 install --no-index osmium --find-links /io && cd /io/test && python3 -m nose && python3 -V"

But after thinking this through again it's natural that this wheels fail on Alpine Linux - it uses libmusl instead of glibc.

[lonvia]

Do binary wheels of other python packages work on Alpine Linux?

[wiktorn]

As far as I've checked - no (officially). They may work if you install glibc on Alpine Linux and set your installation as manylinux compatibile (see pypa/pip#3969). So I guess it's not a showstopper.

I guess that for Alpine it would be the best to provide apk binary package.

[lonvia]

The latest 2.15.0 release now ships with binary wheels. Many thanks to @wiktorn for pushing this forward.