Upload to PyPI

[mumrah]

Before publishing the real thing, push to the staging repository and make sure things look good.

See: https://wiki.python.org/moin/TestPyPI

[zroadhouse-rmn]

+1 - would like to see this project in PyPI

[zroadhouse-rmn]

Though as a temporary workaround, I was able to add a dependency on kafka-python via adding the following to my setup.py:

setup(...
        install_requires=[
                'kafka-python==0.9.0'
        ],
        dependency_links=[
                'https://github.com/mumrah/kafka-python/tarball/master#egg=kafka-python-0.9.0'
        ])

[dsully]

+1 - please publish to PyPi

[sholsapp]

+1. Want any help with this? Want to use Travis CI?

[mumrah]

I think we were holding off on a PyPI upload until we had gotten through a few bugs. However, it might just be best to bite the bullet and upload it.

+1 to upload
+1 to TravisCI doing it

@sholsapp we'd certainly appreciate the help.

@dpkp @mrtheb @mahendra @stephenarmstrong: any objections?

[wizzat]

You should not upload it until it supports 0.8.1 offsets.

[sholsapp]

@wizzat what issue(s) is tracking 0.8.1 offsets?

[wizzat]

Offset support is present in the following pull requests/issues:
#158
#160
#154
#133
#118

[tkuhlman]

Any progress on this? Of the listed pull requests still open a couple have seen no activity for months and the other (154) is reportedly fixed. I can look into the travis-ci changes if that is all that is holding this up.

[dpkp]

I believe the tagged version 0.9.1 should be uploaded to PyPi (https://github.com/mumrah/kafka-python/releases/tag/v0.9.1).

OffsetCommits are now supported and the remaining PRs in that list are either referencing larger issues (related to multiplexing functionality based on server version) or were waiting verification and probably just need to be closed.

I also think we should have travis automatically upload any tagged release to PyPi. Any help on that front would be greatly appreciated!

[mumrah]

+1 to travis auto-releasing tags

[tkuhlman]

I added in the pypi bits for travis here, #198

Pypi needs a user/pass, travis allows you to encrypt it with a command like 'travis encrypt --add deploy.password' details on setting up that travis command are at http://docs.travis-ci.com/user/encryption-keys/
I have a special pypi user that I use for this and I set things up with that user. Creating your own user like that then editing the .travis.yml with that user name and encrypted password is a good path. If you would rather get it done quicker you can just add my user 'tkuhlman-travis' as a maintainer for the project in pypi.

Once things are setup you can go to travis.ci and run a new build.

[dpkp]

the current pypi package is at https://pypi.python.org/pypi/kafka-python

it lists a package index owner of jasontrost . @mumrah do you have maintainer auth for the package? pypi is rejecting my attempts to register.

[mumrah]

I can't seem to update the index either. I've opened a support ticket with PyPI https://sourceforge.net/p/pypi/support-requests/410/

Once we get the package ownership fixed, I'll update the travis info with my credentials.

[dpkp]

I emailed the jason trost on github - https://github.com/jt6211 ... perhaps he can transfer

[mumrah]

I also just emailed the owner of the forked one on PyPI, https://pypi.python.org/pypi/kafka-quixey/0.9.0-q2.

[mumrah]

Yikes, between these two unofficial packages there were 7431 downloads last month o_o

[tkuhlman]

pypi registers downloads immediately after pushing a package, I'm not sure exactly but it seems like there are a lot of pypi mirrors or something along that lines that account for a chunk of those downloads.

Of course some are quite legitimate, anytime anyone reloads our dev environment (https://github.com/stackforge/monasca-vagrant) it pulls it down also.

[mumrah]

@tkuhlman yea I assume about 90% of these kinds of stats are from automated build/deployment scripts, but hey, if 700 people "pip install kafka-python" in a month, I'd be delighted 😄

[jatrost]

I will gladly hand over control of that pypi repo. Just want these guys to confirm their pypi usernames.

[mumrah]

@mumrah is mumrah on PyPI

Thanks @jt6211!

[jatrost]

just added you as an owner

[jatrost]

once you confirm you have access, feel free to remove me as owner.

[mumrah]

OK, I have uploaded the v0.9.1 tag to the test PyPI site: https://testpypi.python.org/pypi/kafka-python/0.9.1. Can a few folks test it out before I push to the real thing?

pip install --index=https://testpypi.python.org/pypi kafka-python

I also have some small changes to setup.py, should I just update the tag?

[tkuhlman]

The test location worked in my install.

Adding a new tag is the best test that Travis CI will pick up the tag and automatically push it to pypi, so putting a 0.9.2 tag with the setup.py updates sounds like a good idea.

[dpkp]

also as soon as the tag is applied, it gets "published" on github . so 0.9.1 already exists here: https://github.com/mumrah/kafka-python/releases/tag/v0.9.1

i think it is better to tag an 0.9.2 release and push that to pypi.

But also for 0.9.2 I would highly recommend merging PR 194 as a quick bugfix and PR 196 for the increased test coverage and socket connection cleanup.

[mumrah]

I'm testing out the PyPI auto-publishing as well real quick. Hopefully https://travis-ci.org/mumrah/kafka-python/builds/33191103 will publish to the PyPI test index.

@dpkp I made a release branch off your tag for this exercise. If you guys want to cut releases from tags on master, that's fine by me, but it will mean that we'll have to hold more closely to "master is always releasable". I tend to prefer release branches, but I don't want to put extra process in places where it isn't needed. WDYT?

[mumrah]

Well I've been trying this all day, and still no luck. I have gotten from Invalid Password to "Upload failed (503): All backends failed or unhealthy" which seems to be an improvement. Will continue trying in the morning :(

[mumrah]

Ok, finally got the upload working to the test PyPI site. Apparently there is a bug with TravisCI and "$" in passwords. @dpkp @wizzat @mrtheb what would you guys like to see for a release process? Just a tag on master, or tags on release branches?

If we did a release branch, I think we should keep it simple, like:

master
|
|
start 0.9 release
|\
|  \
|    \
|    tag 0.9.1, published to PyPI
|     |
|-----* merge/pick from master
|     |
|-----*
|     |
|    tag 0.9.2, published to PyPI

[dpkp]

i generally prefer just committing to the master branch. most folks that submit PRs are going to submit them against master. I dont feel like master needs to always be in a "release" state -- that's what tagging is for, imho. Tagged releases get listed automatically here: https://github.com/mumrah/kafka-python/releases so they should be easy to find. and as long as pypi is only getting uploaded when there is a tag applied, the general pypi public shouldn't care whether master is functional or not.

But that's just my humble opinion, albeit heavily influenced by the chef cookbook development model :)

[tkuhlman]

I generally prefer master also, though have worked with various branching models so whichever the primary devs prefer I will work with.

[mumrah]

Alright, it should now deploy tags on master (or any branch actually). @dpkp you mentioned a few PRs you'd like to incorporate before 0.9.2, so I'll leave it to you to do the release.

Here is the version that I got deployed to the test PyPI site https://testpypi.python.org/pypi?:action=display&name=kafka-python&version=0.9.1.1 btw

Thanks, everyone!

[dpkp]

my testing suggests that travis is going to try to deploy to pypi on every job (currently we have 12), not just once at the end of the build. assuming we only tag after master has successfully built / passed, that might not be horrible process wise. hopefully pypi wont get upset with us resubmitting the same tarball 12x times....

[dpkp]

0.9.1 uploaded to PyPi (previously tagged version). tagging 0.9.2 now

[dpkp]

https://pypi.python.org/pypi/kafka-python