2018.11.14 regression: UnicodeEncodeError: 'ascii' codec can't encode character u'\u2718' in position 0: ordinal not in range(128)

[jfly]

When running pipenv install --system --deploy with pipenv 2018.11.14 on Ubuntu 16.04, I see the following crash:

Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 7, in <module>
    from pipenv import cli
  File "/usr/local/lib/python2.7/dist-packages/pipenv/__init__.py", line 47, in <module>
    from .cli import cli
  File "/usr/local/lib/python2.7/dist-packages/pipenv/cli/__init__.py", line 3, in <module>
    from .command import cli
  File "/usr/local/lib/python2.7/dist-packages/pipenv/cli/command.py", line 18, in <module>
    from .. import environments
  File "/usr/local/lib/python2.7/dist-packages/pipenv/environments.py", line 278, in <module>
    PIPENV_SPINNER_FAIL_TEXT = fs_str(to_text(u"✘ {0}")) if not PIPENV_HIDE_EMOJIS else ("{0}")
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/vistir/compat.py", line 182, in fs_str
    return string.encode(_fs_encoding)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2718' in position 0: ordinal not in range(128)

(What follows is some light investigation)

This new PIPENV_SPINNER_FAIL_TEXT variable seems to have been introduced in 4dac167. The crash is happening in fs_str here:

pipenv/pipenv/vendor/vistir/compat.py

Lines 174 to 182 in d9ebad3

	def fs_str(string):
	"""Encodes a string into the proper filesystem encoding
	Borrowed from pip-tools
	"""
	if isinstance(string, str):
	return string
	assert not isinstance(string, bytes)
	return string.encode(_fs_encoding)

. This crash seems to be happening because sys.getfilesystemencoding() returns "ANSI_X3.4-1968" on Ubuntu 16.04:

➜  ~ docker run -it ubuntu:16.04
root@72aab1a9c1e6:/# apt-get update && apt-get install python 
...
root@72aab1a9c1e6:/# python
Python 2.7.12 (default, Nov 12 2018, 14:36:49) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; print(sys.getfilesystemencoding())
ANSI_X3.4-1968
>>> u'\u2718'.encode("ANSI_X3.4-1968")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2718' in position 0: ordinal not in range(128)

Hope this helps!

[frostming]

We are sorry for the trouble. As a workaround, you can set PIPENV_HIDE_EMOJIS=1.

[techalchemy]

What is your locale setting? Do you have LANG set? I think I’m using a better approach in most places...

[techalchemy]

Seriously need to consolidate the output code

[jfly]

This is an unaltered ubuntu:16.04 image from dockerhub, so you should be able to play with this locally if you've got docker! Some info for you:

➜  ~ docker run -it ubuntu:16.04
root@8aa33a1ac5e7:/# echo $LANG

root@8aa33a1ac5e7:/# env
HOSTNAME=8aa33a1ac5e7
TERM=xterm
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
SHLVL=1
HOME=/root
_=/usr/bin/env
root@8aa33a1ac5e7:/# locale -a
C
C.UTF-8
POSIX
root@8aa33a1ac5e7:/# locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

[techalchemy]

Until this release did this work at all?

[techalchemy]

Context: pipenv has required LANG to be set forever as far as I recall...

[jfly]

Yeah, things worked on pipenv 2018.10.13 and before.

Working

➜  ~/tmp/pipenv-3223 cat Dockerfile
FROM ubuntu:16.04

RUN apt-get update && apt-get install -y python python-pip

RUN pip install pipenv==2018.10.13
➜  ~/tmp/pipenv-3223 docker run $(docker build -q .) pipenv
Usage: pipenv [OPTIONS] COMMAND [ARGS]...
...

Broken

➜  ~/tmp/pipenv-3223 cat Dockerfile 
FROM ubuntu:16.04

RUN apt-get update && apt-get install -y python python-pip

RUN pip install pipenv==2018.11.14
➜  ~/tmp/pipenv-3223 docker run $(docker build -q .) pipenv
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 7, in <module>
    from pipenv import cli
  File "/usr/local/lib/python2.7/dist-packages/pipenv/__init__.py", line 47, in <module>
    from .cli import cli
  File "/usr/local/lib/python2.7/dist-packages/pipenv/cli/__init__.py", line 3, in <module>
    from .command import cli
  File "/usr/local/lib/python2.7/dist-packages/pipenv/cli/command.py", line 18, in <module>
    from .. import environments
  File "/usr/local/lib/python2.7/dist-packages/pipenv/environments.py", line 278, in <module>
    PIPENV_SPINNER_FAIL_TEXT = fs_str(to_text(u"✘ {0}")) if not PIPENV_HIDE_EMOJIS else ("{0}")
  File "/usr/local/lib/python2.7/dist-packages/pipenv/vendor/vistir/compat.py", line 182, in fs_str
    return string.encode(_fs_encoding)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2718' in position 0: ordinal not in range(128)

[techalchemy]

(Waiting on a clean build of that PR but eventually it should be the right approach)