Universal wheel #162

fredrikaverpil · 2016-11-22T06:19:26Z

Edit 1: posted this prematurely (hotkey on commute...), please refresh!
Edit 2: structured up findings into one single post

Goal

Offer one single, universal, wheel for Qt.py: Qt.py-0.6.6-py2.py3-none-any.whl.

Background

I'm maintaining an internal wheel "registry" at my company and I recently realized there are one wheel for each major version of Python:

Qt.py-0.6.6-py2-none-any.whl
Qt.py-0.6.6-py3-none-any.whl

If you attempt to install e.g. the py3 version into Python 2.x you'll get this:

pip2 install Qt.py-0.6.6-py3-none-any.whl
Qt.py-0.6.6-py3-none-any.whl is not a supported wheel on this platform.

So far, I haven't looked into the pip deployment at all, but there seems to be a possibility to have Qt.py use one and the same wheel for either version, which would be nice. An example of another open source package doing this is virtualenv. Here you can see only one wheel is available and it's called virtualenv-15.1.0-py2.py3-none-any.whl.

Implementation

It seems we need to do two things:

Add setup.cfg with the following contents:

[bdist_wheel]
universal=1

Change this line of the Travis PyPi deployment into:

distributions: "sdist bdist_wheel"

Sources:

@mottosso I believe you have a better overall understanding (than me) on how the PyPi deployment works and how to test it out. Would you like to have a look at this? (there's absolutely no hurry on my part on getting this implemented)

The text was updated successfully, but these errors were encountered:

mottosso · 2016-11-22T10:10:31Z

Thanks @fredrikaverpil, looks interesting. I have very little experiences with wheels, my understanding was that they are better (only?) suited for packages that contain binaries, and offer a way of providing identical sources with unique binaries per platform.

Since Qt.py has no binaries, I didn't think it applies.

But, having had a look at the wheels documentation it appears they do apply, and what we're looking for is - as you say - a "universal wheel".

As for testing it out, you should be able to locally build a wheel and see how it works for you.

$ git clone
$ cd Qt.py
$ python setup.py bdist_wheel --universal

It should build a "universal" wheel for you, that you can later try out.

To have it automatically built and uploaded to PyPI, we'll have to pass through Travis, and some quick Googling didn't reveal how to accomplish this.

Simply adding bdist_wheel to our .travis.yml seem to only produce a "Pure Python Wheel", which isn't cross-compatible with Python 3.

From the docs, it might be a combination of bdist_wheel and a new setup.cfg file that does the trick.

I'd be happy for you to try, the only way we can really be sure is by making a few releases and seeing what travis and PyPI says. If you can build it locally first and make sure there isn't anything in Qt.py that hinders the ability, that would be best.

fredrikaverpil · 2016-11-23T14:49:00Z

Sounds good, I'll look into this.

I just noticed these are the variants which can get downloaded from pip wheel -w . Qt.py:

Qt.py-0.6.6-py2-none-any.whl  # macOS
Qt.py-0.6.6-py3-none-any.whl  # macOS
Qt.py-0.6.6-cp27-none-any.whl   # Windows
Qt.py-0.6.6-cp35-none-any.whl  # Windows

...which indicates greater "fragmentation" on the Windows platform, as the minor Python version number is specified in the filename.

fredrikaverpil · 2016-11-23T15:01:36Z

Ok, here we go:

$ python2 --version
Python 2.7.12

$ python2 setup.py bdist_wheel --universal
running bdist_wheel
running build
running build_py
creating build
creating build/lib
copying Qt.py -> build/lib
installing to build/bdist.macosx-10.11-x86_64/wheel
running install
running install_lib
creating build/bdist.macosx-10.11-x86_64
creating build/bdist.macosx-10.11-x86_64/wheel
copying build/lib/Qt.py -> build/bdist.macosx-10.11-x86_64/wheel
running install_egg_info
running egg_info
creating Qt.py.egg-info
writing Qt.py.egg-info/PKG-INFO
writing top-level names to Qt.py.egg-info/top_level.txt
writing dependency_links to Qt.py.egg-info/dependency_links.txt
writing manifest file 'Qt.py.egg-info/SOURCES.txt'
reading manifest file 'Qt.py.egg-info/SOURCES.txt'
writing manifest file 'Qt.py.egg-info/SOURCES.txt'
Copying Qt.py.egg-info to build/bdist.macosx-10.11-x86_64/wheel/Qt.py-0.6.6-py2.7.egg-info
running install_scripts
creating build/bdist.macosx-10.11-x86_64/wheel/Qt.py-0.6.6.dist-info/WHEEL

This resulted in:

$ tree -hs dist
dist
└── [6.4K]  Qt.py-0.6.6-py2.py3-none-any.whl

0 directories, 1 file

Tried to pip install it:

$ pip2 --version
pip 9.0.1 from /usr/local/lib/python2.7/site-packages (python 2.7)

$ pip2 install dist/Qt.py-0.6.6-py2.py3-none-any.whl 
Processing ./dist/Qt.py-0.6.6-py2.py3-none-any.whl
Installing collected packages: Qt.py
Successfully installed Qt.py-0.6.6

$ pip3 --version
pip 9.0.1 from /usr/local/lib/python3.5/site-packages (python 3.5)

$ pip3 install dist/Qt.py-0.6.6-py2.py3-none-any.whl
Processing ./dist/Qt.py-0.6.6-py2.py3-none-any.whl
Installing collected packages: Qt.py
Successfully installed Qt.py-0.6.6

What do you think, should I create a PR and we can try this out with Travis?

All I did was the two things outlined in my initial post:

Add setup.cfg
Add bdist_wheel to Travis config (although this didn't do anything locally)

mottosso · 2016-11-24T09:58:52Z

Yes, put together a PR. We'll have to make releases as well, to get it to upload to PyPI so that you can test. It's safe to delete a release after it's been made, but we can't remove the versions from PyPI, so maybe make the version number something temporary, like 0.6.7-test-01. Then when we remove them and make a final version, we can call it 0.6.7 as per usual.

The reason being that PyPI doesn't allow duplicate versions, at all. So again, don't test with a real version number, or else we'll get a gap in our releases - like 0.6.6 to 0.6.12, once we're done testing and removing releases. Not the end of the world, but if it can be avoided, let's.

Does that make sense?

fredrikaverpil · 2016-11-24T11:42:59Z

Yes, makes a lot of sense. I'd like to go through this together with you thoroughly, since like you say, it's irreversible with PyPi. I'll set up a PR and we'll continue talking there before actually tagging.

fredrikaverpil · 2016-11-24T17:57:09Z

I made a release (0.6.7, fixing the QtCompat attributes) which turned up almost instantly in PyPi. Then I created release 0.6.8-test-01 but I still don't see that in PyPi.

I wonder if I did things in the wrong order... although I followed your instructions. Where can you review the deployment status?

fredrikaverpil · 2016-11-24T18:07:06Z

Oh, this sucks / learning new things here!

HTTPError: 400 Client Error: Invalid version, cannot be parsed as a valid PEP 440 version. for url: https://pypi.python.org/pypi
/home/travis/virtualenv/python2.7.10/lib/python2.7/site-packages/setuptools/dist.py:294: UserWarning: The version specified ('0.6.8-test-01') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.

fredrikaverpil · 2016-11-24T18:47:13Z

Gah, managed to create the 0.6.8.dev1 release before I merged #166...

Okay, let's have another try with 0.6.8.dev2 and #167.

fredrikaverpil · 2016-11-24T19:01:05Z

I keep getting this:

Skipping a deployment with the pypi provider because this is not a tagged commit

I don't understand how to fullfil that requirement. I do a merge with a version bump in Qt.py and then I create a new release with the corresponding tag/version. What am I missing?

fredrikaverpil · 2016-11-24T19:08:00Z

I keep getting this

Ah, I was looking at the wrong commit, it seems. When looking at the correct one it now seems like it worked fine (although a small error which we've probably had all along).

Uploading distributions to https://pypi.python.org/pypi
Uploading Qt.py-0.6.8.dev2.tar.gz
[================================] 9578/9578 - 00:00:03
running upload_docs
creating build
creating build/docs
error: no files found in upload directory 'build/docs'

No stash found.

Done. Your build exited with 0.

EDIT:

PyPi was updated with the release too: https://pypi.python.org/pypi/Qt.py/0.6.8.dev2

But I don't see the new wheel just yet:

$ pip wheel --no-cache -w . Qt.py
Collecting Qt.py
  Downloading Qt.py-0.6.7.tar.gz
Building wheels for collected packages: Qt.py
  Running setup.py bdist_wheel for Qt.py ... done
  Stored in directory: /Users/fredrik/code
Successfully built Qt.py

fredrikaverpil · 2016-11-24T19:14:58Z

Pip doesn't download the tar.tgz file yet, butI can download it manually from PyPi. From that I can generate a universal wheel which can be installed in Python 2.7 and 3.5:

$ pip wheel -w . /Users/fredrik/Downloads/Qt.py-0.6.8.dev2.tar.gz 
Processing /Users/fredrik/Downloads/Qt.py-0.6.8.dev2.tar.gz
Building wheels for collected packages: Qt.py
  Running setup.py bdist_wheel for Qt.py ... done
  Stored in directory: /Users/fredrik/code
Successfully built Qt.py

$ ll *.whl
-rw-r--r--  1 fredrik  staff   6.5K Nov 24 20:13 Qt.py-0.6.8.dev2-py2.py3-none-any.whl

$ pip2 install Qt.py-0.6.8.dev2-py2.py3-none-any.whl 
Processing ./Qt.py-0.6.8.dev2-py2.py3-none-any.whl
Installing collected packages: Qt.py
Successfully installed Qt.py-0.6.8.dev2

$pip3 install Qt.py-0.6.8.dev2-py2.py3-none-any.whl 
Processing ./Qt.py-0.6.8.dev2-py2.py3-none-any.whl
Installing collected packages: Qt.py
Successfully installed Qt.py-0.6.8.dev2

fredrikaverpil · 2016-11-25T05:20:14Z

Pip still doesn't download the tar.tgz hosted in PyPi. Is it possible this is because it's a dev release?
Either way, it now looks like the wheel is universal and we should be able to file a new non-dev release.

@mottosso What do you think?

mottosso · 2016-11-25T08:52:21Z

That's an interesting feature, looks like releases marked dev have special meaning to pip.

https://packaging.python.org/installing/#installing-prereleases

You install them using the --pre flag.

$ pip install --pre Qt.py

Looks good to me, let's pull the trigger!

fredrikaverpil · 2016-11-25T09:19:57Z

Ah!

That's actually a very nice feature (in our case).

$ pip2 wheel --no-cache --pre -w . Qt.py
Collecting Qt.py
  Downloading Qt.py-0.6.8.dev2.tar.gz
Building wheels for collected packages: Qt.py
  Running setup.py bdist_wheel for Qt.py ... done
  Stored in directory: c:\users\iruser\code\repos\qt.py
Successfully built Qt.py

$ dir *.whl
 Volume in drive C is Windows
 Volume Serial Number is 92FF-FCC6

 Directory of C:\Users\iruser\code\repos\Qt.py

11/25/2016  10:18 AM             6,723 Qt.py-0.6.8.dev2-py2.py3-none-any.whl
               1 File(s)          6,723 bytes
               0 Dir(s)  1,807,888,773,120 bytes free

$ pip2 install Qt.py-0.6.8.dev2-py2.py3-none-any.whl
Processing c:\users\iruser\code\repos\qt.py\qt.py-0.6.8.dev2-py2.py3-none-any.whl
Installing collected packages: Qt.py
Successfully installed Qt.py-0.6.8.dev2

$ pip3 install Qt.py-0.6.8.dev2-py2.py3-none-any.whl
Processing c:\users\iruser\code\repos\qt.py\qt.py-0.6.8.dev2-py2.py3-none-any.whl
Installing collected packages: Qt.py
Successfully installed Qt.py-0.6.8.dev2

So, what do you say, should I just change the Qt.py version into 0.6.8 and create a new 0.6.8 release to make this happen?

fredrikaverpil · 2016-11-25T10:00:42Z

Allright, implemented in #168 🎉

Qt.py-0.6.8-py2.py3-none-any.whl

mottosso · 2016-11-25T10:02:00Z

Whop whop! Good job!

greatestape · 2017-01-29T16:57:39Z

Hi everyone. I discovered your repo by chance while trying to solve the universal wheel issue for my project. It looks like we both have the same bug: we put the "distributions" field inside the "on" part of "deploy".

Looking back at the comments here, it looks like travis was never actually building the wheel for you, and I think this is why.

I've submitted a pull request with the fix. I know it's hard to test changes to the "deploy" part of a travis config so you might be hesitant to change it, but I thought I'd offer up this fix anyway, because I found this to be a hard bug to track down.

fredrikaverpil added the enhancement label Nov 22, 2016

fredrikaverpil changed the title ~~Python version-agnostic wheel~~ Universal wheel Nov 22, 2016

fredrikaverpil mentioned this issue Nov 24, 2016

Feature/universal wheel #163

Merged

This was referenced Nov 24, 2016

Universal wheel: attempt abide PEP440 #166

Merged

Universal wheel: Bump dev version #167

Merged

fredrikaverpil mentioned this issue Nov 25, 2016

Bump version for universal wheel #168

Merged

fredrikaverpil closed this as completed in #168 Nov 25, 2016

greatestape mentioned this issue Jan 29, 2017

Move "distributions" out of "on" part of "deploy" #178

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Universal wheel #162

Universal wheel #162

fredrikaverpil commented Nov 22, 2016 •

edited

Loading

mottosso commented Nov 22, 2016

fredrikaverpil commented Nov 23, 2016 •

edited

Loading

fredrikaverpil commented Nov 23, 2016 •

edited

Loading

mottosso commented Nov 24, 2016

fredrikaverpil commented Nov 24, 2016

fredrikaverpil commented Nov 24, 2016 •

edited

Loading

fredrikaverpil commented Nov 24, 2016 •

edited

Loading

fredrikaverpil commented Nov 24, 2016 •

edited

Loading

fredrikaverpil commented Nov 24, 2016

fredrikaverpil commented Nov 24, 2016 •

edited

Loading

fredrikaverpil commented Nov 24, 2016 •

edited

Loading

fredrikaverpil commented Nov 25, 2016 •

edited

Loading

mottosso commented Nov 25, 2016

fredrikaverpil commented Nov 25, 2016 •

edited

Loading

fredrikaverpil commented Nov 25, 2016

mottosso commented Nov 25, 2016

greatestape commented Jan 29, 2017

Universal wheel #162

Universal wheel #162

Comments

fredrikaverpil commented Nov 22, 2016 • edited Loading

Goal

Background

Implementation

mottosso commented Nov 22, 2016

fredrikaverpil commented Nov 23, 2016 • edited Loading

fredrikaverpil commented Nov 23, 2016 • edited Loading

mottosso commented Nov 24, 2016

fredrikaverpil commented Nov 24, 2016

fredrikaverpil commented Nov 24, 2016 • edited Loading

fredrikaverpil commented Nov 24, 2016 • edited Loading

fredrikaverpil commented Nov 24, 2016 • edited Loading

fredrikaverpil commented Nov 24, 2016

fredrikaverpil commented Nov 24, 2016 • edited Loading

fredrikaverpil commented Nov 24, 2016 • edited Loading

fredrikaverpil commented Nov 25, 2016 • edited Loading

mottosso commented Nov 25, 2016

fredrikaverpil commented Nov 25, 2016 • edited Loading

fredrikaverpil commented Nov 25, 2016

mottosso commented Nov 25, 2016

greatestape commented Jan 29, 2017

fredrikaverpil commented Nov 22, 2016 •

edited

Loading

fredrikaverpil commented Nov 23, 2016 •

edited

Loading

fredrikaverpil commented Nov 23, 2016 •

edited

Loading

fredrikaverpil commented Nov 24, 2016 •

edited

Loading

fredrikaverpil commented Nov 24, 2016 •

edited

Loading

fredrikaverpil commented Nov 24, 2016 •

edited

Loading

fredrikaverpil commented Nov 24, 2016 •

edited

Loading

fredrikaverpil commented Nov 24, 2016 •

edited

Loading

fredrikaverpil commented Nov 25, 2016 •

edited

Loading

fredrikaverpil commented Nov 25, 2016 •

edited

Loading