pipenv install blows up with a TypeError

[Julian]

Issue description

Running pipenv install foo blows up with bytes being passed around where unicode is expected.

Expected result

foo is installed.

Actual result

Traceback (most recent call last):
  File "/Users/Julian/.local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/cli.py", line 435, in install
    selective_upgrade=selective_upgrade,
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/core.py", line 2046, in do_install
    skip_lock=skip_lock
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/core.py", line 1312, in do_init
    pypi_mirror=pypi_mirror,
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/core.py", line 1140, in do_lock
    project.write_lockfile(lockfile)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/project.py", line 651, in write_lockfile
    f.write(s)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/_compat.py", line 216, in func_wrapper
    return func(*args, **kwargs)
TypeError: unicode argument expected, got 'str'

Steps to replicate

⊙  rm -rf ~/.local/share/virtualenvs/Foo-mHoil3nZ && mkdir Foo && cd Foo && echo "[packages]" >Pipfile && pipenv install requests                                                                                                                                                                                                                         Julian@Macnetic ●
Installing requests...
Collecting requests
  Using cached https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.8,>=2.5 (from requests)
  Using cached https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl
Collecting urllib3<1.24,>=1.21.1 (from requests)
  Using cached https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
  Using cached https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl
Installing collected packages: chardet, idna, urllib3, certifi, requests
Successfully installed certifi-2018.4.16 chardet-3.0.4 idna-2.7 requests-2.19.1 urllib3-1.23

Adding requests to Pipfile's [packages]...
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Traceback (most recent call last):
  File "/Users/Julian/.local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/cli.py", line 435, in install
    selective_upgrade=selective_upgrade,
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/core.py", line 2046, in do_install
    skip_lock=skip_lock
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/core.py", line 1312, in do_init
    pypi_mirror=pypi_mirror,
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/core.py", line 1140, in do_lock
    project.write_lockfile(lockfile)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/project.py", line 651, in write_lockfile
    f.write(s)
  File "/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/_compat.py", line 216, in func_wrapper
    return func(*args, **kwargs)
TypeError: unicode argument expected, got 'str'
/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv/_compat.py:113: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/kh/mscfrt910qb_qb8wqrz7t_7w0000gn/T/pipenv-C4owd0-requirements'>
  warnings.warn(warn_message, ResourceWarning)

Instrumenting pipenv.project to just have it print the reprs of what it's trying to do shows that it's trying to call simplejson.dumps on

{u'_meta': {u'sources': [{u'url': u'https://pypi.org/simple', u'verify_ssl': True, u'name': u'pypi'}], u'requires': {}, u'hash': {u'sha256': u'70e8bf6bc774f5ca177467cab4e67d4264d0536857993326abc13ff43063bec0'}, u'pipfile-spec': 6}, u'default': {u'certifi': {'version': '==2018.4.16', 'hashes': [u'sha256:13e698f54293db9f89122b0581843a782ad0934a4fe0172d2a980ba77fc61bb7', u'sha256:9fa520c1bacfb634fa7af20a76bcbd3d5fb390481724c597da32c719a7dca4b0']}, u'chardet': {'version': '==3.0.4', 'hashes': [u'sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae', u'sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691']}, u'idna': {'version': '==2.7', 'hashes': [u'sha256:156a6814fb5ac1fc6850fb002e0852d56c0c8d2531923a51032d1b70760e186e', u'sha256:684a38a6f903c1d71d6d5fac066b58d7768af4de2b832e426ec79c30daa94a16']}, u'requests': {'version': '==2.19.1', 'hashes': [u'sha256:63b52e3c866428a224f97cab011de738c36aec0185aa91cfacd418b5d58911d1', u'sha256:ec22d826a36ed72a7358ff3fe56cbd4ba69dd7a6718ffd450ff0e9df7a47ce6a'], 'index': u'pypi'}, u'urllib3': {'version': '==1.23', 'hashes': [u'sha256:a68ac5e15e76e7e5dd2b8f94007233e01effe3e50e8daddf69acfd81cb686baf', u'sha256:b5725a0bd4ba422ab0e66e89e030c806576753ea3ee08554382c14e685d117b5'], 'markers': "python_version >= '2.6' and python_version != '3.0.*' and python_version != '3.1.*' and python_version != '3.2.*' and python_version != '3.3.*' and python_version < '4'"}}, u'develop': {}}

producing

'{\n    "_meta": {\n        "hash": {\n            "sha256": "70e8bf6bc774f5ca177467cab4e67d4264d0536857993326abc13ff43063bec0"\n        },\n        "pipfile-spec": 6,\n        "requires": {},\n        "sources": [\n            {\n                "name": "pypi",\n                "url": "https://pypi.org/simple",\n                "verify_ssl": true\n            }\n        ]\n    },\n    "default": {\n        "certifi": {\n            "hashes": [\n                "sha256:13e698f54293db9f89122b0581843a782ad0934a4fe0172d2a980ba77fc61bb7",\n                "sha256:9fa520c1bacfb634fa7af20a76bcbd3d5fb390481724c597da32c719a7dca4b0"\n            ],\n            "version": "==2018.4.16"\n        },\n        "chardet": {\n            "hashes": [\n                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",\n                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"\n            ],\n            "version": "==3.0.4"\n        },\n        "idna": {\n            "hashes": [\n                "sha256:156a6814fb5ac1fc6850fb002e0852d56c0c8d2531923a51032d1b70760e186e",\n                "sha256:684a38a6f903c1d71d6d5fac066b58d7768af4de2b832e426ec79c30daa94a16"\n            ],\n            "version": "==2.7"\n        },\n        "requests": {\n            "hashes": [\n                "sha256:63b52e3c866428a224f97cab011de738c36aec0185aa91cfacd418b5d58911d1",\n                "sha256:ec22d826a36ed72a7358ff3fe56cbd4ba69dd7a6718ffd450ff0e9df7a47ce6a"\n            ],\n            "index": "pypi",\n            "version": "==2.19.1"\n        },\n        "urllib3": {\n            "hashes": [\n                "sha256:a68ac5e15e76e7e5dd2b8f94007233e01effe3e50e8daddf69acfd81cb686baf",\n                "sha256:b5725a0bd4ba422ab0e66e89e030c806576753ea3ee08554382c14e685d117b5"\n            ],\n            "markers": "python_version >= \'2.6\' and python_version != \'3.0.*\' and python_version != \'3.1.*\' and python_version != \'3.2.*\' and python_version != \'3.3.*\' and python_version < \'4\'",\n            "version": "==1.23"\n        }\n    },\n    "develop": {}\n}'

which is a str (this is Py2).

Note that things like the version key there in the dict has bytes as its value, not unicode (and despite a comment in the code there saying " # Send Unicode in to guarentee Unicode out.", which the code at least locally there seems to expect to be receiving).

(Possible aside: from just glancing at that function, it seems like it's suffering from trying to use unicode on both Py2 and Py3, rather than using native strings on Py2 and Py3, which IME is way easier to do when dealing with files.)

---

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/Users/Julian/.local/share/virtualenvs/pipenv/site-packages/pipenv'

Python location: '/Users/Julian/.local/share/virtualenvs/pipenv/bin/python'

Other Python installations in PATH:

• 2.7: /usr/local/bin/python2.7

• 2.7: /usr/local/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.7: /usr/local/bin/python3.7m

• 3.7: /usr/local/bin/python3.7

• 2.7.13: /Users/Julian/.local/bin/python

• 2.7.15: /usr/local/bin/python

• 2.7.10: /usr/bin/python

• 2.7.15: /Users/Julian/.local/bin/python2

• 2.7.15: /usr/local/bin/python2

• 3.7.0: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'PyPy',
 'platform_release': '17.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.6.0: Tue May  8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASE_X86_64',
 'python_full_version': '2.7.13',
 'python_version': '2.7',
 'sys_platform': 'darwin'}

System environment variables:

• ANSIBLE_CONFIG
• ANSIBLE_HOSTS
• ANSIBLE_RETRY_FILES_SAVE_PATH
• ANSIBLE_VAULT_PASSWORD_FILE
• AWS_CONFIG_FILE
• AWS_SHARED_CREDENTIALS_FILE
• Apple_PubSub_Socket_Render
• BOTO_CONFIG
• BROWSER
• BUNDLE_CONFIG
• CENTOS_7_AMI
• CHEF_PATH
• COLORFGBG
• COLORTERM
• COMMAND_MODE
• CTAGS
• DELETE_ON_TERMINATION
• DEVELOPMENT
• DISPLAY
• DOCKER_CONFIG
• EDITOR
• GEM_HOME
• GEM_SPEC_CACHE
• GIT_TEMPLATE_DIR
• GNUPGHOME
• GPG_TTY
• HOME
• HTTPIE_CONFIG_DIR
• INPUTRC
• IS_DEVELOPMENT_WORKSTATION
• ITERM_PROFILE
• ITERM_SESSION_ID
• LANGUAGE
• LEIN_HOME
• LESSHISTFILE
• LOGNAME
• LS_COLORS
• M2_HOME
• MAGNETIC_DEVELOPMENT
• MYSQL_HISTFILE
• NOTMUCH_CONFIG
• NPM_CONFIG_USERCONFIG
• NVIM_TUI_ENABLE_CURSOR_SHAPE
• OLDPWD
• PATH
• PERL5LIB
• PERL_CPANM_HOME
• PERL_LOCAL_LIB_ROOT
• PERL_MB_OPT
• PERL_MM_OPT
• PIP_CONFIG_FILE
• PSQL_HISTORY
• PWD
• PYTHONDONTWRITEBYTECODE
• PYTHONSTARTUP
• PYTHONWARNINGS
• PYTHON_TEST_RUNNER
• SECURITYSESSIONID
• SHELL
• SHLVL
• SSH_AUTH_SOCK
• STARDICT_DATA_DIR
• TASKRC
• TERM
• TERM_PROGRAM
• TERM_PROGRAM_VERSION
• TERM_SESSION_ID
• TMPDIR
• TMUX
• TMUX_PANE
• USER
• VIRTUALENVS
• XDG_CACHE_HOME
• XDG_CONFIG_HOME
• XDG_DATA_HOME
• XDG_DESKTOP_DIR
• XDG_DOCUMENTS_DIR
• XDG_DOWNLOAD_DIR
• XDG_MUSIC_DIR
• XDG_PICTURES_DIR
• XDG_PUBLICSHARE_DIR
• XDG_TEMPLATES_DIR
• XDG_VIDEOS_DIR
• XINITRC
• XPC_FLAGS
• XPC_SERVICE_NAME
• ZDOTDIR
• ZSHPLUGINS
• __CF_USER_TEXT_ENCODING
• C_INCLUDE_PATH
• CPLUS_INCLUDE_PATH
• GOPATH
• LIBRARY_PATH
• LD_LIBRARY_PATH
• PKG_CONFIG_PATH
• NODEPATH
• PYTHONPATH
• HGRCPATH
• WORKON_HOME
• LANG
• LC_ALL
• _
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /Users/Julian/.local/bin:/usr/local/bin:/usr/local/sbin:/Users/Julian/.local/share/gem/bin:/Users/Julian/Library/Application Support/go/bin:/usr/local/share/pypy:/usr/local/share/pypy3:/usr/local/share/npm/bin:/usr/local/opt/coreutils/libexec/gnubin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Applications/Wireshark.app/Contents/MacOS:/Users/Julian/Library/Application Support/perl5/bin
• SHELL: /usr/local/bin/zsh
• EDITOR: nvim
• LANG: en_US.UTF-8
• PWD: /Users/Julian/Desktop

---

[techalchemy]

the question is really, how did the key come to be written as non-unicode? But we can fix this by doing better assurances . While it is easier to simply use strings, that is not really a good approach. Unicode exists for a lot of reasons and we do absolutely need to use it.

[uranusjr]

IMO the correct way to achieve Python 2/3 support is to always use Unicode, not str. Unicode everywhere makes the code much easier to reason with.

The comment in the dumps line (I wrote that)

s = simplejson.dumps(  # Send Unicode in to guarentee Unicode out.
    content, indent=4, separators=(u",", u": "), sort_keys=True
)

shows that we (I?) were very concious about always using Unicode, just not doing it correctly.

[Julian]

I'm not a maintainer so I guess I won't argue much about the implementation, whatever is convenient for y'all, though yes I certainly do disagree :).

Anything else I can provide here to help track this down?

[techalchemy]

Hmmmm did you already have a pipfile here? If so we might be a bit Unicode aggressive as you say. We are supposed to respect existing pipfile encoding

[Julian]

So, that command above is a reproducer on this machine (rm -rf ~/.local/share/virtualenvs/Foo-mHoil3nZ && mkdir Foo && cd Foo && echo "[packages]" >Pipfile && pipenv install requests), and which is there outputting a tiny Pipfile (a minimal reproducer even), but I notice that from another
macOS box set up essentially identically I don't get an error, so will work on figuring out what part of env is likely to be involved here.

[Julian]

OK, this still fails with env -i, so whatever it is isn't something harmful in the environment... more digging.

[techalchemy]

You have your locale set?

[Julian]

Normally yes (though with env -i it's unset).

I just found the problematic code though, it's here:

https://github.com/pypa/pipenv/blob/master/pipenv/utils.py#L1262

Where that's assembling a dict with str values, which then the code in my traceback is assuming is unicode.

[avhn]

I had the same problem. Can we fix it in package too?
If you use OS X, add these to ~/.bash_profile

export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8

Then
$ source ~/.bash_profile

Edit: I guess it's about encoding of interaction between python and shell? How can we solve this permanently within the pipenv?

Source: https://stackoverflow.com/a/38917471/9201345

[Julian]

My locale is appropriately set, see the original env dump at the top.

The code I linked doesn't depend on the locale anyhow.

[techalchemy]

Thanks for the debugging effort, and yeah, makes sense to me. For that I guess you’d need PYTHONIOENCODING if we don’t set it. We should set it correctly.

[uranusjr]

I’m updating #2561 to also ensure the lock file gets unicode to write on Python 2. This should be fixed after it’s merged.

Getting this identical problem with most recent release, but I only get it when using pyenv.