Can't lock package with `git` and `extras`

[JacobHayes]

Issue description

A package spec with git and extras fails to lock. This happens regardless of the editable flag.

Expected result

The package should be installed with all extras.

Actual result

[I] ~/test $ pipenv --version
pipenv, version 2018.7.1
[I] ~/test $ ll
total 8
-rw-r--r--  1 jacobhayes    96B Jul  3 13:12 Pipfile
[I] ~/test $ cat Pipfile
[packages]
requests = {git = "https://github.com/requests/requests.git", extras = ["security"]}
[I] ~/test $ pipenv --rm ; pipenv install --verbose
Removing virtualenv (/Users/jacobhayes/.local/share/virtualenvs/test-qFl43Lkt)...
Creating a virtualenv for this project...
Pipfile: /Users/jacobhayes/test/Pipfile
Using /usr/local/opt/python/bin/python3.6 (3.6.5) to create virtualenv...
⠋Already using interpreter /usr/local/opt/python/bin/python3.6
Using base prefix '/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/jacobhayes/.local/share/virtualenvs/test-qFl43Lkt/bin/python3.6
Also creating executable in /Users/jacobhayes/.local/share/virtualenvs/test-qFl43Lkt/bin/python
Installing setuptools, pip, wheel...done.
Setting project for test-qFl43Lkt to /Users/jacobhayes/test

Virtualenv location: /Users/jacobhayes/.local/share/virtualenvs/test-qFl43Lkt
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 435, in install
    selective_upgrade=selective_upgrade,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1943, in do_install
    pypi_mirror=pypi_mirror,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1312, in do_init
    pypi_mirror=pypi_mirror,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1140, in do_lock
    project.write_lockfile(lockfile)
  File "/usr/local/lib/python3.6/site-packages/pipenv/project.py", line 648, in write_lockfile
    content, indent=4, separators=(u',', u': '), sort_keys=True,
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 430, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 437, in _iterencode
    o = _default(o)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'ArrayElement' is not JSON serializable

Steps to replicate

[packages]
requests = {git = "https://github.com/requests/requests.git", extras = ["security"]}

pipenv install

---

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/usr/local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/local/opt/python/bin/python3.6'

Other Python installations in PATH:

• 2.7: /usr/local/bin/python2.7

• 2.7: /usr/local/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.6: /usr/local/bin/python3.6m

• 3.6: /usr/local/bin/python3.6

• 2.7.15: /usr/local/bin/python

• 2.7.10: /usr/bin/python

• 2.7.15: /usr/local/bin/python2

• 3.6.5: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.6.0: Tue May  8 15:22:16 PDT '
                     '2018; root:xnu-4570.61.1~1/RELEASE_X86_64',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'darwin'}

System environment variables:

• Apple_PubSub_Socket_Render
• COLORFGBG
• COLORTERM
• COPYFILE_DISABLE
• EDITOR
• GOPATH
• HOME
• ITERM_PROFILE
• ITERM_SESSION_ID
• LANG
• LOGNAME
• PAGER
• PATH
• PWD
• SECURITYSESSIONID
• SHELL
• SHLVL
• SSH_AUTH_SOCK
• TERM
• TERM_PROGRAM
• TERM_PROGRAM_VERSION
• TERM_SESSION_ID
• TMPDIR
• TMUX
• TMUX_PANE
• TMUX_PLUGIN_MANAGER_PATH
• USER
• XPC_FLAGS
• XPC_SERVICE_NAME
• __CF_USER_TEXT_ENCODING
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /Users/jacobhayes/bin:/Users/jacobhayes/bin:/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin:/Users/jacobhayes/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
• SHELL: /usr/local/bin/fish
• EDITOR: vim
• LANG: en_US.UTF-8
• PWD: /Users/jacobhayes/test

---

Contents of Pipfile ('/Users/jacobhayes/test/Pipfile'):

[packages]
requests = {git = "https://github.com/requests/requests.git", extras = ["security"]}

[JacobHayes]

This works with v2018.05.18, but fails in v2018.6.25 and v2018.7.1

[techalchemy]

Interesting! I can definitely replicate this.

[theunraveler]

git bisect tells me that 18530df is the culprit, if that helps.

[bryanforbes]

I did some digging today and an easy fix for this is to make the following change in Project.write_lockfile():

    def write_lockfile(self, content):
        """Write out the lockfile.
        """
        newlines = self._lockfile_newlines

        def encode_objects(obj):
            if isinstance(obj, ArrayElement): # prettytoml.elements.array.ArrayElement
                return obj.primitive_value
            return obj

        s = simplejson.dumps(   # Send Unicode in to guarentee Unicode out.
            content, indent=4, separators=(u',', u': '), sort_keys=True,
            default=encode_objects
        )
        with atomic_open_for_write(self.lockfile_location, newline=newlines) as f:
            f.write(s)
            if not s.endswith(u'\n'):
                f.write(u'\n')  # Write newline at end of document. GH #319.

[techalchemy]

thanks for digging into this, I know very little about the toml parser -- do you have a sense of what's going on here?

[bryanforbes]

When the lockfile is read from the file, it just copies the structure over from the parsed TOML. I'm not sure why, but the TOML structure returns a specialized ArrayElement instead of a Python list (which is why simplejson.dumps() throws an error). Rather than iterate through the TOML coming out of the parser, it looked easier to have simplejson.dumps() convert the ArrayElements to lists.

[notEvil]

Development since v2018.5.18 completely killed our deployment. This bug and #2317 are easily worked around (patch and external variable substitution) but #2552 is probably not. It would be great to have pipenv tested so that these issues don't find their way to the open :)
Otherwise great work, its much appreciated

[techalchemy]

This bug is a 3.7+ bug and should only be happening since the combination of 3.7 + 2018.7.1. I took the weekend off of working on this project, but we’d definitely merge a failing test / fix if we haven’t already (not totally caught up here)

[theunraveler]

This bug is a 3.7+ bug and should only be happening since the combination of 3.7 + 2018.7.1.

This is definitely happening on 3.6.* for me.

[JacobHayes]

@bryanforbes would you be willing to submit a PR with your changes or mind if I do?

[bryanforbes]

@JacobHayes you're more than welcome to do the PR. I'm swamped this week.