Switching to pre-releases instead of cirq-unstable

[balopat]

Switch to PyPi pre-releases instead of cirq-unstable.

Two major reasons for this change:

1. in preparation for modules - for which we would have to duplicate all of them if we followed this pattern.
2. As raised in Prevent installing both cirq and cirq-unstable. #3267, cirq-unstable doesn't play well with cirq installations

Fixes #3267.

A resulting state after a test-run for this looks like this on the Test PyPi:

The dev releases will still get the "this is an unstable version" notice for their description.

[viathor]

LGTM with minor comments.

[viathor]

IIUC, this should be changed: "As it contains" -> "If it contains"

("As" implies we know it does, which I don't think we do.)

[balopat]

We can clarify that - but it is assumed that it contains dev.

[balopat]

Actually it can contain whatever, but it is assumed that it contains devN, aN, bN, rcN hence the long description that changes it to the warning text below

[balopat]

Thanks @viathor - I'm going to have to put a pause on this as I discovered Craig's comment here, that pointed out that tensorflow and tfq-nightly have the same issue as cirq and cirq-unstable: #2431 (comment).
Also I need a bit more evidence that in a multi-module setting the pre-releases will play nicely together - i.e. whether I can setup for example pip install --pre cirq-pasqal should bring in the correct latest cirq... I'll convert this back to live if I'm convinced that this is the right way to go.

[mpharrigan]

Isn't Craig's comment further justification for using pre-releases?

[balopat]

Isn't Craig's comment further justification for using pre-releases?

I interpreted it as "if this is okay with tensorflow, it should be okay for us too to live with this" based on the context that he seemed to be "pro separate package".

[balopat]

Oh by the way some more updates on this:

1. https://www.python.org/dev/peps/pep-0440/#developmental-releases:

Note

While they may be useful for continuous integration purposes, publishing developmental releases of pre-releases to general purpose public index servers is strongly discouraged, as it makes the version identifier difficult to parse for human readers. If such a release needs to be published, it is substantially clearer to instead create a new pre-release by incrementing the numeric component.

Developmental releases of post-releases are also strongly discouraged, but they may be appropriate for projects which use the post-release notation for full maintenance releases which may include code changes.

2. There is no impact on dependency management whether we use the same package or separate ("unstable") packages. It is straightforward to generate requirements to the given version in the setup.py for each module.

I'm more concerned about the 1st point and playing nice in the python community. More motivating conversations around this:

• How PyPi does not allow for retention policy: https://discuss.python.org/t/publishing-nightly-builds-on-test-pypi-org-with-a-time-based-retention-policy/3152
• scikiy-learn ended up on anaconda.org as they have the ability to delete packages: https://docs.anaconda.com/anaconda-cloud/user-guide/tasks/work-with-packages/

The only issue I have with anaconda.org is that this is how we'd install modules:

pip install --pre --extra-index https://pypi.anaconda.org/cirq-unstable/simple cirq

pip install --pre --extra-index https://pypi.anaconda.org/cirq-unstable/simple cirq-google

[balopat]

Oh by the way some more updates on this:

1. https://www.python.org/dev/peps/pep-0440/#developmental-releases:

Note
While they may be useful for continuous integration purposes, publishing developmental releases of pre-releases to general purpose public index servers is strongly discouraged, as it makes the version identifier difficult to parse for human readers. If such a release needs to be published, it is substantially clearer to instead create a new pre-release by incrementing the numeric component.
Developmental releases of post-releases are also strongly discouraged, but they may be appropriate for projects which use the post-release notation for full maintenance releases which may include code changes.

1. There is no impact on dependency management whether we use the same package or separate ("unstable") packages. It is straightforward to generate requirements to the given version in the setup.py for each module.

I'm more concerned about the 1st point and playing nice in the python community. More motivating conversations around this:

• How PyPi does not allow for retention policy: https://discuss.python.org/t/publishing-nightly-builds-on-test-pypi-org-with-a-time-based-retention-policy/3152
• scikiy-learn ended up on anaconda.org as they have the ability to delete packages: https://docs.anaconda.com/anaconda-cloud/user-guide/tasks/work-with-packages/

The only issue I have with anaconda.org is that this is how we'd install modules:

pip install --pre --extra-index https://pypi.anaconda.org/cirq-unstable/simple cirq

pip install --pre --extra-index https://pypi.anaconda.org/cirq-unstable/simple cirq-google

Oh hold on - the first point is irrelevant, it is just a strong recommendation around non-human-readable version numbers. Ignore that.

I was for a second also worried about "littering" the main package with dev releases for readability - but maybe it's not that big of a deal. Also, pre-releases like v1.0.0.rc1, v1.0.0.rc2 can still work, just pip install --pre cirq will be shadowed by the dev releases, so one will have to use the specific versions there instead explicitly.
2. In terms of space, I am actually not that worried about it. With 1.6MB per release and ~800 commits per year we are below a 1GB per year.

So the only question remains is readability of different versions on the cirq pypi page...

[balopat]

In the end I am ready to merge this as it will make our (less packages to deal with) and our users lives easier (no weird namespace collisions).

As I understand the per project limit is 10GB which we'll get to in 5+ years if we never clean up a release.
Cleaning up releases is manual at this point - but there is an issue for PyPi that pushes for automated cleanups (pypi/warehouse#8792).

@quantumlib/cirq-maintainers this is a final chance to object :) Please comment, otherwise I'm ready to merge tomorrow!

[mpharrigan]

All in all, I think this is a good idea. We did this for a previous open source project I maintained and it worked well iirc

[maffoo]

Minor comment about release docs, otherwise this LGTM!

[balopat]

As we have quorum, I'm going to go ahead and merge it. Thanks all for chiming in!