Deployment questions

[joerick]

Questions from @zfrenchee in #4 - I'm posting these to a new thread for better visibility

1. You say in the "Manual Deploy" instructions:

# Go and download your wheel files from wherever you put them.

but I'm not sure in my current config where I'm even putting my wheels. My mental model for travis is that it's spinning up a VM, doing all these tasks, then throwing away all the data and reporting back whether it succeeded. It's putting all the wheels in wheelhouse/ but I'm not sure how to access that directory, and I'm assuming it's removed once the travis job completes. Is that mental model incorrect? How would I go about downloading my wheels from wheelhouse/?

2. Wheelhouse uploader, which you recommend as the semi-automatic method, seems to mainly be built to upload wheels to clouds like s3, and then perhaps download them later to be uploaded to pypi. But it seems like s3 buckets are required to make it work. Am I reading that wrong -- Is there a way to use wheelhouse uploader to upload wheels from the travis VM?

[joerick]

So, in your first question -

I'm not sure in my current config where I'm even putting my wheels

currently, Travis CI offer no way to archive your build artifacts themselves. Your mental model is correct - you have no direct access to the VM, and it's destroyed once the build finishes. Therefore, to get hold of the build wheels, you have to upload them to a temporary location. Travis have a number of options for places you can upload to. (It's much easier on Appveyor - they retain build artifacts for you).

I already use Amazon S3 for other things, so I set my builds to upload there. Rackspace's Cloud Files is also popular for this kind of thing.

Is there a way to use wheelhouse uploader to upload wheels from the travis VM?

yep. wheelhouse-uploader is a tool that semi-automates the deploy process. Deployment to PyPI is still manual, but the download of the built wheels from cloud storage is just one command. Note that you'll still need a cloud provider as in the previous example. As a bonus you also get an index file that pip can install wheels from, before releasing to PyPI.

There are other options, such as uploading to Github releases, or directly to PyPI. Both require you to tag a release before it's built. You get less control over the release, but you don't need the file hosting service to get the job done.

[alexlenail]

@joerick I'm interested in deploying directly to PyPi, despite the loss of some control. I just can't seem to figure out how. Would you be willing to help me out, that way we can document this strategy for future users of cibuildwheel who also want to just have an easy way to get all their wheels on pypi in one step?

[alexlenail]

I'm also getting this other issue with wheelhouse uploader: ogrisel/wheelhouse-uploader#21

[joerick]

@zfrenchee I'll see if I can get get something working 👍

[alexlenail]

@joerick I've managed to get deployment up and working for linux, but not macOS.

It looks like despite building wheels successfully, the deploy option tries to build wheels again (but only on macOS)

I asked Travis support about this, and here's what I got back:

Thanks for following up. This is indeed a bit curious. As Hiro already noted, Python is not really supported on OSX, as also noted in https://docs.travis-ci.com/user/languages/python/#What-This-Guide-Covers.

We suspect that the error relates to the difference in virtual environment setup, as we have it in Linux, and the non-venv setup on OSX. This means that the $PATH environment variable is not set up as the pip deployment expects, which results in that the twine package cannot be installed.

Maybe you would be better off when you explicitly create a virtual environment for your project, and source bin/activate in your build.

You might need to only execute this step on OSX, which may be achieved through:
- if [ $TRAVIS_OS_NAME = osx ]; then source bin/activate; fi

I hope this helps to get you on the right track.

I think this might be something worth resolving in cibuildwheel?

[joerick]

It looks like Travis' PyPI integration tries to build distributions before uploading from the dist folder. I'm looking at adding a few lines to travis YML to use twine directly. It should copy/paste straight into travis yml without much extra config

[joerick]

Hi @zfrenchee, check out https://github.com/joerick/cibuildwheel-autopypi-example/blob/master/.travis.yml - it seems to be working well for me and isn't a big addition to the travis.yml 😄

[alexlenail]

Hi @joerick. As usual, thank you so much for this support! What is TRAVIS_TAG in the example you shared? I adopted the code from above and it "ran" but doesn't seem to have uploaded any mac wheels, but I didn't explicitly set TRAVIS_TAG anywhere.

[joerick]

TRAVIS_TAG is a variable set by TravisCI when it's building a Git tag, so it only triggers the wheel upload when you've pushed a tagged commit.