New pip 9 breaks install of pyasn1 #276

jecevans · 2016-11-19T17:03:33Z

While trying to build a new Octopi image on Rasbian I get the following ouput.

Collecting pyasn1
Downloading pyasn1-0.1.9-py2.py3-none-any.whl
Requirement already satisfied: PyOpenSSL in ./oprint/lib/python2.7/site-packages (from ndg-httpsclient)
Exception:
Traceback (most recent call last):
File "/home/pi/oprint/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/pi/oprint/local/lib/python2.7/site-packages/pip/commands/install.py", line 324, in run
requirement_set.prepare_files(finder)
File "/home/pi/oprint/local/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/home/pi/oprint/local/lib/python2.7/site-packages/pip/req/req_set.py", line 666, in _prepare_file
check_dist_requires_python(dist)
File "/home/pi/oprint/local/lib/python2.7/site-packages/pip/utils/packaging.py", line 48, in check_dist_requires_python
feed_parser.feed(metadata)
File "/usr/lib/python2.7/email/feedparser.py", line 177, in feed
self._input.push(data)
File "/usr/lib/python2.7/email/feedparser.py", line 99, in push
parts = data.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'

The version I am building off of is
Linux Rasbian 4.4.32-v7+ #924 SMP Tue Nov 15 18:11:28 GMT 2016 armv7l GNU/Linux

ScottDWebster · 2016-11-20T01:56:01Z

I am also seeing this behavior. I've tried building on raspbian (2016-09-23-raspian-jessie.zip) and from ubuntu 16.04.1 x86. In both cases I am trying to build an octopi image based on the full raspbian jessie with pixel so I can also make use of the desktop features on the machine as well.

At various times during the attempted build, I see a message like "sudo: unable to resolve host ubuntu-too" but "ping ubuntu-too" finds the correct ip address.
octopi-build.log.txt

guysoft · 2016-11-21T13:42:12Z

Confirmed,
First build this started happening at Build #779 (Nov 3, 2016 2:00:01 AM)
Can compare to working build the day before: Build #778 (Nov 2, 2016 2:00:01 AM)

guysoft · 2016-11-21T13:42:27Z

cc @foosel

guysoft · 2016-11-21T14:40:59Z

Ran a diff against the build outputs. The only diffrence I can spot is that during pip upgrade:

Installing collected packages: pip
  Found existing installation: pip 1.5.6
    Uninstalling pip:
      Successfully uninstalled pip
Compiling /tmp/pip-build-ReWmM6/pip/pip/_vendor/html5lib/treewalkers/etree.py ...
  File "/tmp/pip-build-ReWmM6/pip/pip/_vendor/html5lib/treewalkers/etree.py", line 13
    from pip._vendor. import string_types
                           ^
SyntaxError: invalid syntax

foosel · 2016-11-21T14:43:10Z

Looks like a bug that only occurs with current pip version 9.x (which gets installed a couple steps prior).

I can reproduce (even on an older raspbian I might add) via:

sudo apt-get install python-openssl
virtualenv test
./test/bin/pip install --upgrade pip
for pattern in "pyOpenSSL*"; do cp -v -r --preserve=mode,timestamps /usr/lib/python2.7/dist-packages/$pattern test/lib/python2.7/site-packages/; done
./test/bin/pip install ndg-httpsclient

I cannot reproduce any more after downgrading pip to 8.1.2:

./test/bin/pip install --upgrade "pip>=8,<9"
./test/bin/pip install ndg-httpsclient

Looks like pip 9.x hiccups on the .egg-info file from the PyOpenSSL debian package, or rather doesn't know what to do with it since it expects newer kinds of metadata.

That whole mess with copying python ssl stuff back and forth was added in order to get a sane SSL environment in urllib3/requests under the (old) python version shipped with raspbian previously (see here for details). What's the currently available one? If it's >= 2.7.9, it might be possible to get rid of that whole madness which would be good.

guysoft · 2016-11-21T14:49:45Z

This is the bug in pip: pypa/pip#4059 fixed in 9.0.1

foosel · 2016-11-21T15:06:10Z

No. That's the issue you saw on your Jenkins, but I didn't see it when testing locally and neither did the log provided by @ScottDWebster contain that issue.

That's not what's breaking installation of ndg-httpsclient here (pyasn1 installs fine btw, it's indeed ndg-httpsclient in the same pip call as asn1 that's causing the problem).

guysoft · 2016-11-21T15:29:41Z

Just ran a build with your fix and it fixes it: http://gnethomelinux.com:8080/job/OctoPi/800/console
@jecevans and @ScottDWebster , Pull from devel and you can confirm

guysoft · 2016-11-21T15:31:04Z

Note: we will need to revert this when pip 9.0.1 comes out.

foosel · 2016-11-21T15:33:16Z

Pinning pip to version 8 is NOT a fix. It's a workaround, and pip 9.0.1 is already out and does also show that behaviour. pip will keep on bugging people to upgrade it, and that might then cause issues in the field.

This needs to be solved either by getting rid of that whole manual SSL setup for Python (which again is not necessary IF the current Raspbian ships with a Python version >= 2.7.9 OR at least allows upgrading to one, someone please test this, I can't right now!) or if that's not possible finding a (probably more resource intense) solution to the old pyopenssl debian package causing this issue with current pip versions.

You need to solve the problem, not patch up the symptom.

guysoft · 2016-11-21T15:41:31Z

@foosel See the issue i posted from pip, its a vendor inside pip which I don't thing we can really control. One alternative would be to pull from https://github.com/pypa/pip/archive/9.0.1.zip .
Trying that now. I guess I am guilty of wanting to fix this quickly because I have a bit of load at work.

foosel · 2016-11-21T16:10:47Z

Again, there are TWO issues at work here, ~~and 9.0.1 is affected for both~~ and the more severe one affects 9.0.1 just as much:

That vendor thing, that doesn't appear to have any real consequences though, other than ugly log messages, and it only occurs in your Jenkins log, it is not happening in the log provided by @ScottDWebster
The issue quoted in the OP, the Traceback caused during install of pyasn1 and ndg-httpsclient (with the latter being the actual culprit, pip install pyasn1 runs flawlessly). THAT actually has consequences, making the build fail.

Both issues are completely unrelated! One (the vendor thing) is an actual pip issue, but rather harmless. The other is a compatibility issue between the version of PyOpenSSL manually installed during an earlier step of the build script and pip versions >= 9.0 (including 9.0.1, which is already released and failing as well, see my posts above), and what's the actual problem that needs to be solved here.

Since current Python version on Raspbian finally seems to be 2.7.9, try removing these lines from the build script INSTEAD of the downgrade/version pin of pip (so making this only read pip install --upgrade pip again). Try that, see if the build a) succeeds and b) when actually running the image OctoPrint doesn't drown the log in "insecure platform warning" messages.

guysoft · 2016-11-21T16:20:39Z

Th full log os not provided. Would be helpful if OP provides it. Not
sure about @ScottDWebster 's log.
They must be related in some way. When I change the pip line the pyasn1
stops breaking. And it was consist.

On Nov 21, 2016 18:10, "Gina Häußge" notifications@github.com wrote:

Again, there are TWO issues at work here, and 9.0.1 is affected for both:

That vendor thing, that doesn't appear to have any real
consequences though, other than ugly log messages, and it only occurs in
your Jenkins log, it is not happening in the log provided by
@ScottDWebster https://github.com/ScottDWebster

The issue quoted in the OP, the Traceback caused during install of
pyasn1 and ndg-httpsclient (with the latter being the actual culprit, pip
install pyasn1 runs flawlessly). THAT actually has consequences,
making the build fail.

Both issues are completely unrelated! One (the vendor thing) is an actual
pip issue, but rather harmless. The other is a compatibility issue between
the version of PyOpenSSL manually installed during an earlier step of the
build script and pip versions >= 9.0 (including 9.0.1, which is already
released and failing, see my posts above), and what's the actual problem
that needs to be solved here.

Since current Python version on Raspbian finally seems to be 2.7.9, try
removing these lines
https://github.com/guysoft/OctoPi/blame/devel/src/chroot_script#L71-L91
from the build script INSTEAD of the downgrade/version pin of pip (so
making this only read pip install --upgrade pip again). Try that, see if
the build a) succeeds and b) when actually running the image OctoPrint
doesn't drown the log in "insecure platform warning" messages.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
#276 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAT4Ji-JslmS4xSgCFBPFgI-NlcJ4Zucks5rAcKIgaJpZM4K3Tdt
.

foosel · 2016-11-21T16:26:56Z

They are both not present in pip 8.1.2, which is what gets installed when you specify to install a pip version >= 8 and < 9 (your commit comment is also wrong in that regard, you pinned the install to 8.1.2, not 9.0.0 like the comment suggests - 9.0.0 would be ">=8,<=9.0.0"). That's the only way they are related, if you can call that related.

Case in point: http://gnethomelinux.com:8080/job/OctoPi/798/console. Installs 9.0.1, no vendor error:

02:09:57 Downloading/unpacking pip from https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
02:09:59 Installing collected packages: pip
02:09:59   Found existing installation: pip 1.5.6
02:09:59     Uninstalling pip:
02:09:59       Successfully uninstalled pip
02:10:06 Successfully installed pip

Still breaks on the ndg-httpsclient line:

02:12:26 Requirement already satisfied: PyOpenSSL in ./oprint/lib/python2.7/site-packages (from ndg-httpsclient)
02:12:26 Exception:
02:12:26 Traceback (most recent call last):
02:12:26   File "/home/pi/oprint/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
02:12:26     status = self.run(options, args)
02:12:26   File "/home/pi/oprint/local/lib/python2.7/site-packages/pip/commands/install.py", line 324, in run
02:12:26     requirement_set.prepare_files(finder)
02:12:26   File "/home/pi/oprint/local/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
02:12:26     ignore_dependencies=self.ignore_dependencies))
02:12:26   File "/home/pi/oprint/local/lib/python2.7/site-packages/pip/req/req_set.py", line 666, in _prepare_file
02:12:26     check_dist_requires_python(dist)
02:12:26   File "/home/pi/oprint/local/lib/python2.7/site-packages/pip/utils/packaging.py", line 48, in check_dist_requires_python
02:12:26     feed_parser.feed(metadata)
02:12:26   File "/usr/lib/python2.7/email/feedparser.py", line 177, in feed
02:12:26     self._input.push(data)
02:12:26   File "/usr/lib/python2.7/email/feedparser.py", line 99, in push
02:12:26     parts = data.splitlines(True)
02:12:26 AttributeError: 'NoneType' object has no attribute 'splitlines'

Two issues. Not one. The vendor thing is a red herring.

ScottDWebster · 2016-11-21T16:35:01Z

By the way, "python -V" on the 2016-09-23 Raspbian Jessie (albeit an dist-upgraded version) says "Python 2.7.9".

…(OctoPi)

We no longer need to manually setup a solid SSL environment since that's already done for use with Python 2.7.9. The whole issue that we ran into with setting this up under Pip 9+ therefore can be circumvented without having to pin the pip version. Refer to guysoft#276

Visdoom · 2017-08-07T09:02:41Z

Hey guys,
any news on this?

I got the same error when trying to get
pip install imbalanced-learn to work.

Error log:

Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.4/dist-packages/pip/commands/install.py", line 335, in run
    wb.build(autobuilding=True)
  File "/usr/local/lib/python3.4/dist-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/local/lib/python3.4/dist-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/local/lib/python3.4/dist-packages/pip/req/req_set.py", line 666, in _prepare_file
    check_dist_requires_python(dist)
  File "/usr/local/lib/python3.4/dist-packages/pip/utils/packaging.py", line 48, in check_dist_requires_python
    feed_parser.feed(metadata)
  File "/usr/lib/python3.4/email/feedparser.py", line 177, in feed
    self._input.push(data)
  File "/usr/lib/python3.4/email/feedparser.py", line 100, in push
    parts = data.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'

Downgrading to pip 8.1.2 fixed it

guysoft · 2017-08-07T12:33:20Z

There is nothing we can really do here other than downgrading to pip 8.1.2. Someone would need to send a pull request to pip's team.

guysoft · 2019-04-23T08:51:08Z

Fixed in fb7093b

guysoft added the bug label Nov 21, 2016

guysoft changed the title ~~Attempting to build new image on Rasbian failed~~ New pip 9 breaks install of pyasn1 Nov 21, 2016

guysoft closed this as completed in dc66324 Nov 21, 2016

guysoft reopened this Nov 21, 2016

david-pacheco added a commit to beeverycreative/BEEwebPi that referenced this issue Nov 28, 2016

Workaround pip 9.0.0 that breaks install of pyasn1 fixes guysoft#276 …

4c7dddb

…(OctoPi)

foosel mentioned this issue Feb 28, 2017

Remove pip version pin #315

Merged

guysoft mentioned this issue Mar 31, 2017

OctoPi 0.14.0 RC1 status #332

Closed

guysoft mentioned this issue Jun 18, 2017

OctoPi 0.14.0 RC2 status #367

Closed

guysoft mentioned this issue Jun 25, 2017

OctoPi 0.14.0 RC3 status #371

Closed

markwal mentioned this issue Jun 25, 2017

Install fails during dependency checks... markwal/OctoPrint-PolarCloud#1

Closed

susannasiebert mentioned this issue Sep 12, 2017

Installation error off of develop branch griffithlab/pVACtools#16

Closed

adam-iris mentioned this issue Apr 11, 2018

programmatic install quiet option fails pypa/pip#4694

Closed

guysoft closed this as completed Apr 23, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New pip 9 breaks install of pyasn1 #276

New pip 9 breaks install of pyasn1 #276

jecevans commented Nov 19, 2016

ScottDWebster commented Nov 20, 2016

guysoft commented Nov 21, 2016 •

edited

Loading

guysoft commented Nov 21, 2016

guysoft commented Nov 21, 2016

foosel commented Nov 21, 2016

guysoft commented Nov 21, 2016 •

edited

Loading

foosel commented Nov 21, 2016

guysoft commented Nov 21, 2016

guysoft commented Nov 21, 2016 •

edited

Loading

foosel commented Nov 21, 2016

guysoft commented Nov 21, 2016

foosel commented Nov 21, 2016 •

edited

Loading

guysoft commented Nov 21, 2016

foosel commented Nov 21, 2016 •

edited

Loading

ScottDWebster commented Nov 21, 2016

Visdoom commented Aug 7, 2017

guysoft commented Aug 7, 2017

guysoft commented Apr 23, 2019

New pip 9 breaks install of pyasn1 #276

New pip 9 breaks install of pyasn1 #276

Comments

jecevans commented Nov 19, 2016

ScottDWebster commented Nov 20, 2016

guysoft commented Nov 21, 2016 • edited Loading

guysoft commented Nov 21, 2016

guysoft commented Nov 21, 2016

foosel commented Nov 21, 2016

guysoft commented Nov 21, 2016 • edited Loading

foosel commented Nov 21, 2016

guysoft commented Nov 21, 2016

guysoft commented Nov 21, 2016 • edited Loading

foosel commented Nov 21, 2016

guysoft commented Nov 21, 2016

foosel commented Nov 21, 2016 • edited Loading

guysoft commented Nov 21, 2016

foosel commented Nov 21, 2016 • edited Loading

ScottDWebster commented Nov 21, 2016

Visdoom commented Aug 7, 2017

guysoft commented Aug 7, 2017

guysoft commented Apr 23, 2019

guysoft commented Nov 21, 2016 •

edited

Loading

guysoft commented Nov 21, 2016 •

edited

Loading

guysoft commented Nov 21, 2016 •

edited

Loading

foosel commented Nov 21, 2016 •

edited

Loading

foosel commented Nov 21, 2016 •

edited

Loading