pip installation dependencies inconsistency

[samskillman]

Hi,

When pip installing version 0.24.0, I think there is an inconsistency in the dependencies with google-cloud-error-reporting. In the 0.24.0 tagged release, which is what pip install google-cloud picks up, the setup.py associated with it has google-cloud-error-reporting >=0.24.0,<0.25dev. See https://github.com/GoogleCloudPlatform/google-cloud-python/blob/0.24.0/setup.py

Because google-cloud-error-reporting cut a new release of 0.24.1 (see https://pypi.python.org/pypi/google-cloud-error-reporting), which itself requires google-cloud-logging > 1.0.0 (https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/error_reporting/setup.py#L55) it creates an inconsistency with the google-cloud-logging > 0.24.0,<0.25dev (https://github.com/GoogleCloudPlatform/google-cloud-python/blob/0.24.0/setup.py#L60).

While pip install google-cloud functions correctly and you'll be able to import all the packages and use them until there is an api inconsistency, if you have a script that is installed that requires google-cloud, pkg_resources will fail. This comes up in packages that I build that install a script through setup.py in the scripts=[] section. It installs the script and at the top puts something like:

__requires__ = 'my_package'
__import__('pkg_resources').require('my_package')

When this gets executed and my_package requires google-cloud, then the pkg_resources for google-cloud get checked.

To reproduce on python 2.7.6:

pip install google-cloud==0.24.0

python -c "__requires__=['google-cloud']; import pkg_resources"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3036, in <module>
    @_call_aside
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3020, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3049, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 656, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 669, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 859, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (google-cloud-logging 0.24.0 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('google-cloud-logging<2.0dev,>=1.0.0'), set(['google-cloud-error-reporting']))

I realize this is a super hard thing to guard against with so many sub-packages and inter-dependencies, but maybe a test that just does the above requires and import pkg_resources on previous pip-installed tagged versions would help? Maybe you have something like this already on master, since if you manually clone the google-cloud-python repository and run setup.py develop or install, the above pkg_resources error does not show up.

I think a fix might be removing the 0.24.1 google-cloud-error-reporting pypi release and re-releasing as 0.25.0, but I'm not sure that's a great option. Alternatively release a 0.24.1 google-cloud that pegs google-cloud-error-reporting==0.24.0?

[dhermes]

Thanks for reporting @samskillman!

It seems we should have gone to 0.25.0 on error-reporting after the 1.0.0 upgrade for logging (it lagged a little because of a difficult bug).

@lukesneeringer We should cut a new release of google-cloud to eliminate this contradiction.

[lukesneeringer]

Yep. Agreed.

Another thing we should do (I will change this) is have CI run the unit tests as a group, in a joint virtualenv, rather than as individual packages, which would catch this error. Of course, that has some risk -- there could be another error with running them independently that this would miss. Perhaps it would be possible to do a mix of both.

We can think through that later. Can you cover doing the updated release?

[scjody]

Note that this prevents installation of google-cloud using pip-compile: jazzband/pip-tools#497

[dhermes]

@lukesneeringer I'll do this release along with #3332

[lukesneeringer]

Thanks!

[samskillman]

Confirmed this was fixed by the bump to 0.25.0. Thanks!

[nhammond]

This bug broke 0.24.0 when 0.25.0 was released. Then it broke 0.25.0 when 0.26.0 was released. Then it broke 0.26.0 when 0.27.0 was released. I don't think this is resolved. If I used 0.27.0 I fully expect my install to be broken when 0.28.0 comes out.

Currently broken: 0.24.0, 0.25.0, 0.26.0
Currently work: 0.24.1, 0.25.1, 0.26.1, 0.27.0

Reproduction steps:

docker run -it --rm ubuntu bash
apt-get update
apt-get install python-pip
pip install google-cloud==0.26.0
python -c "__requires__=['google-cloud']; import pkg_resources"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2927, in <module>
    @_call_aside
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2913, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2940, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 637, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 650, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 834, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (google-cloud-logging 1.2.0 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('google-cloud-logging<1.2dev,>=1.1.0'), set(['google-cloud-error-reporting']))