in _get_lsb_release_info a lack of a judgment:'utf8' codec can't decode byte 0xd5 in position 24: invalid continuation byte

[lionor666]

• Pip version:9.0.1
• Python version:2.7.14
• Operating system:redhat7.2

Description:

None provided.

What I've run:

51_billappxz%pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pip (9.0.1)
setuptools (38.2.4)
Traceback (most recent call last):
  File "/data01/billappxz/scripts/ThirdParty/python27/bin/pip", line 11, in <module>
    load_entry_point('pip==9.0.1', 'console_scripts', 'pip')()
  File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/__init__.py", line 233, in main
    return command.main(cmd_args)
  File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 251, in main
    timeout=min(5, options.timeout)) as session:
  File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 72, in _build_session
    insecure_hosts=options.trusted_hosts,
  File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 329, in __init__
    self.headers["User-Agent"] = user_agent()
  File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 93, in user_agent
    from pip._vendor import distro
  File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/distro.py", line 1050, in <module>
    _distro = LinuxDistribution()
  File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/distro.py", line 594, in __init__
    if include_lsb else {}
  File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/distro.py", line 922, in _get_lsb_release_info
    stdout, stderr = stdout.decode('utf-8'), stderr.decode('utf-8')
  File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd5 in position 24: invalid continuation byte

in the pip/_vendor/distro.py file, the 922 line. There is no judgment of the result of the execution of the command -'lsb_release -a'. If the lsb_release is not installed on the host ,the return result value is 0xd5 .
Using utf8 coding for 0xd5, the error will be reported.

[pradyunsg]

Hi @lionor666!

Your comment seems to be poorly formatted. I request you to read https://help.github.com/articles/getting-started-with-writing-and-formatting-on-github/ and improve the formatting of your comment by editing it in place, using the pencil icon in the corner.

[pradyunsg]

In case someone wants to look at the changes I've made to OP, it's pasted below.

• Pip version:9.0.1
• Python version:2.7.14
• Operating system:redhat7.2

Description:

51_billappxz%pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pip (9.0.1)
setuptools (38.2.4)
Traceback (most recent call last):
File "/data01/billappxz/scripts/ThirdParty/python27/bin/pip", line 11, in
load_entry_point('pip==9.0.1', 'console_scripts', 'pip')()
File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/init.py", line 233, in main
return command.main(cmd_args)
File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 251, in main
timeout=min(5, options.timeout)) as session:
File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 72, in _build_session
insecure_hosts=options.trusted_hosts,
File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 329, in init
self.headers["User-Agent"] = user_agent()
File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 93, in user_agent
from pip._vendor import distro
File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/distro.py", line 1050, in
_distro = LinuxDistribution()
File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/distro.py", line 594, in init
if include_lsb else {}
File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/_vendor/distro.py", line 922, in _get_lsb_release_info
stdout, stderr = stdout.decode('utf-8'), stderr.decode('utf-8')
File "/data01/billappxz/scripts/ThirdParty/python27/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd5 in position 24: invalid continuation byte

What I've run:

pip list

// REPLACE ME: Paste a log of command(s) you ran and pip's output, tracebacks, etc, here

in the pip/_vendor/distro.py file, the 922 line. There is no judgment of the result of the execution of the command -'lsb_release -a'. If the lsb_release is not installed on the host ,the return result value is 0xd5 .
Using utf8 coding for 0xd5, the error will be reported.

[doloopwhile]

I encountered same error.

• Pip: 9.0.1
• Python: 3.6.3
• OS: CentOS 6.3

I think it is a because shell encoding is not 'utf-8' in my system ($LANG=ja_JP.eucJP) and
pip tries to decode non-utf8 bytes with 'utf-8'.

https://github.com/pypa/pip/blob/master/src/pip/_vendor/distro.py#L924

stdout and stderr should decoded with sys.getfilesystemencoding().

So, I submitted a pull request. Could you review it ?

Stdout and stderr of shell should be decoded with system encoding by doloopwhile · Pull Request #4932 · pypa/pip
#4932

[doloopwhile]

I sent a pull request to distro.
Stdout of shell should be decoded with sys.getfilesystemencoding() by doloopwhile · Pull Request #203 · nir0s/distro

[xavfernandez]

This was likely fixed by #4932.

[pradyunsg]

Yep.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.