pip 8.0.0 has been released, breaks new installs

[natefoo]

pip 8.0.0 was released today. "Galaxy pip" is a fork of pip at a point in time of the development of 8.0.0 (and as such carries the version 8.0.0.dev0+gx1). If you attempt to start Galaxy in a fresh venv today, the step that installs Galaxy pip will instead install pip 8.0.0. pip 8.0.0 includes part of our modifications: SOABI tag support for 2.X, but not support for linux platform tag specificity. Thus, we need to update Galaxy pip to the release version of 8.0.0 plus the platform tag specificity modifications to create version 8.0.0+gx1.

We may also want to explicitly install this version in common_startup.sh. The reason this was not done previously is that we can release bug fixes to Galaxy pip that will automatically be installed by common_startup.sh. The downside of this approach is that every time a new version of pip is released, we will need to release a new version of Galaxy pip to supersede it.

[martenson]

will the original pip never sometime adopt the modifications we need?

[natefoo]

It's difficult to say, my previous discussion about it eventually stalled and I haven't yet picked it back up and tried to get some support for it. At a minimum, I believe my platform library would probably need to be spun out into an external project.

But the discussion splintered somewhat with @njsmith advocating for not supporting a wide array of distros as I implemented it, but instead supporting a "baseline linux" platform on which a wheel could be built and then work on an overwhelming majority of installed linux systems. The result of this is the recently announced manylinux and I do think this solves most of the problems we want to solve, probably in a simpler way than I am doing it with the multidistro solution used with our psycopg2 wheels (although it requires either static linking, copying dynamic libs into the wheel, or creating wheels that only contain dynamic libs, which is tricky). And it is roughly the same approach I am using with the rest of our Linux wheels (which are not distro-specific, but assume a certain baseline) but requires that a system explicitly be known to be at that baseline before attempting to install a wheel.

I do have some thoughts on the relative pros and cons between the manylinux approach and my approach and think there's a place for both approaches. The manylinux approach certainly has its advantages, though. The use of my approach is, for example, the reason we haven't yet done anything with galaxyproject/starforge#44, and manylinux would solve that.

I think regardless of what happens with my work, I'd like to support manylinux and see wheels with the manylinux1 platform tag be allowed into PyPI.

[martenson]

Thank you for the great explanation @natefoo !

[peterjc]

i.e. https://www.python.org/dev/peps/pep-0513/

PEP 0513 -- A Platform Tag for Portable Linux Built Distributions

This PEP proposes the creation of a new platform tag for Python package built distributions, such as wheels, called manylinux1{x86_64,i386} with external dependencies limited restricted to a standardized subset of the Linux kernel and core userspace ABI. It proposes that PyPI support uploading and distributing Wheels with this platform tag, and that pip support downloading and installing these packages on compatible platforms._

[natefoo]

Okay, clarification, this does not break new installs because of the way we are installing pip:

$ pip install --pre --no-index --find-links https://wheels.galaxyproject.org/simple/pip --upgrade pip

This prevents pip from being upgraded from PyPI and so as far as this invocation of pip is concerned, pip 8.0.0.dev0+gx1 is the newest version unless you already have 8.0.0 installed.

It breaks Travis because tox creates a new virtualenv which probably contains pip 8.0.0, and then it runs run_tests.sh with --no-create-venv (which is good), but pip cannot be "upgraded" from 8.0.0 to 8.0.0.dev0+gx1 (which is bad). The 8.0.0+gx1 wheel fixes this:

(.venv)nate@weyerbacher% pip install --upgrade pip
Collecting pip
  Downloading pip-8.0.0-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 656kB/s 
Installing collected packages: pip
  Found existing installation: pip 8.0.0.dev0+gx1
    Uninstalling pip-8.0.0.dev0+gx1:
      Successfully uninstalled pip-8.0.0.dev0+gx1
Successfully installed pip-8.0.0
(.venv)nate@weyerbacher% pip install --pre --no-index --find-links http://localhost:8000 --upgrade pip
Ignoring indexes: https://pypi.python.org/simple
Collecting pip
  Downloading http://localhost:8000/pip-8.0.0%2Bgx1-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 91.0MB/s 
Installing collected packages: pip
  Found existing installation: pip 8.0.0
    Uninstalling pip-8.0.0:
      Successfully uninstalled pip-8.0.0
Successfully installed pip-8.0.0+gx1

I have built a pip 8.0.0+gx1 wheel which I'll release on wheels.galaxyproject.org shortly.

[martenson]

nice solution, semantic versioning ftw

[nsoranzo]

pip 8.0.2 is already out, will it break Travis and possibly new installs?

[natefoo]

It will break Travis and any installs where the installed pip version (prior to running common_startup.sh) is > 8.0.0. This would be somewhat mitigated once the automagic wheel update process is running. Duplicating the rest of my comment on galaxyproject/ansible-galaxy#9:

--no-index --find-links https://wheels.galaxyproject.org/simple/pip should prevent it from installing a newer pip from PyPI (unless that newer version is already installed). This was discussed on IRC today since that's exactly what the case will always be with Travis. The proposed solution is an "always install Galaxy pip even if it's a 'downgrade'" option, although I think we should probably make this the default as long as --no-replace-pip is not specified.

That said, I am not sure if it's possible with pip if you don't specify the exact version as part of pip install as you've done here, which I would prefer not to do because I want to be able to release bug fixes.

[nsoranzo]

@natefoo I also cannot find a way to downgrade pip without specifying an exact version. But isn't this the same of pinned requirements for all the other wheels?
I mean, to release bug fixes for pip we would just need to create 2 PRs, one for starforge and one for Galaxy (until that's automated).

[natefoo]

The way things are currently done, we could release a bug fix for pip by simply placing a new version of the pip wheel on wheels.galaxyproject.org. That version would be installed the next time common_startup.sh was run. That said, it's counter to how we release stable fixes to any other part of Galaxy.

Alternatively, we could modify our version to be something like 8.99+8.0.2.gx1 which would cause ours to always be installed but allow for updates, and reflect the upstream version.

[peterjc]

Just FYI, having updated my development instance of Galaxy via git (and thus seeing all the new wheel stuff in one go), I see:

$ ./run.sh
Activating virtualenv at .venv
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Ignoring indexes: https://pypi.python.org/simple
/mnt/galaxy/repositories/galaxy/.venv/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/mnt/galaxy/repositories/galaxy/.venv/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Collecting pip
  Downloading https://wheels.galaxyproject.org/packages/pip-8.0.0+gx1-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 323kB/s 
Installing collected packages: pip
  Found existing installation: pip 8.0.0-gx1
    Uninstalling pip-8.0.0-gx1:
      Successfully uninstalled pip-8.0.0-gx1
Successfully installed pip-8.0.0+gx1
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
...
You are using pip version 8.0.0+gx1, however version 8.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
...

The pip warning (and the Python 2.6 warning) appear multiple times.

So this isn't quite a new install, but it worked despite the pip version warnings.

[natefoo]

Glad to at least see that it worked, thanks @peterjc. I suspect Galaxy itself will drop Python 2.6 support some time this year, although I suppose that might depend on RHEL 7 adoption rates.

[natefoo]

I created an issue regarding Python 2.6 as #1596

[natefoo]

Fixed (again) in #1595