Extras are not really kept across installs

[WhyNotHugo]

Issue description

When installing a package with extras (eg: gunicorn[eventlet]), the extras dependency is installed, but the marker "markers": "extra == 'eventlet'", results it in not being installed when re-creating the environment (eg: pipenv install when the virtualenv does not exist.

Expected result

The extras dependency (in this case, eventlet) should be installed both when initially installing gunicorn[eventlet], and when re creating the environment. The resulting environment should always be consistent.

Actual result

The extra-dependency is only installed when initially installing the depending package, but not when recreating the environment.

Steps to replicate

pipenv install "gunicorn[eventlet]"
pip freeze | grep eventlet  # eventlet is installed
# delete the virtualenv
pipenv install
pip freeze | grep eventlet  # eventlet is NOT installed

---

$ pipenv --support

Pipenv version: '2018.10.13'

Pipenv location: '/usr/local/lib/python3.7/site-packages/pipenv'

Python location: '/usr/local/bin/python'

Python installations found:

• 3.7.0: /usr/local/bin/python3.7
• 3.7.0: /usr/local/bin/python3.7m
• 3.5.3: /usr/bin/python3.5
• 3.5.3: /usr/bin/python3.5m
• 2.7.13: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.18.12-arch1-1-ARCH',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT Thu Oct 4 01:01:27 UTC 2018',
 'python_full_version': '3.7.0',
 'python_version': '3.7',
 'sys_platform': 'linux'}

System environment variables:

• LANG
• HOSTNAME
• GPG_KEY
• PWD
• HOME
• TERM
• PYTHON_VERSION
• SHLVL
• PATH
• PYTHON_PIP_VERSION
• _
• OLDPWD
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
• LANG: C.UTF-8
• PWD: /app

---

Contents of Pipfile ('/app/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
gunicorn = {version = "*", extras = ["eventlet"]}

[dev-packages]

[requires]
python_version = "3.7"

Contents of Pipfile.lock ('/app/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "9b50dce94c5549c220ca646cbf9b03ac24a93703b27aca580f8fe80fdb263e50"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "dnspython": {
            "hashes": [
                "sha256:40f563e1f7a7b80dc5a4e76ad75c23da53d62f1e15e6e517293b04e1f84ead7c",
                "sha256:861e6e58faa730f9845aaaa9c6c832851fbf89382ac52915a51f89c71accdd31"
            ],
            "version": "==1.15.0"
        },
        "eventlet": {
            "hashes": [
                "sha256:c584163e006e613707e224552fafc63e4e0aa31d7de0ab18b481ac0b385254c8",
                "sha256:d9d31a3c8dbcedbcce5859a919956d934685b17323fc80e1077cb344a2ffa68d"
            ],
            "markers": "extra == 'eventlet'",
            "version": "==0.24.1"
        },
        "greenlet": {
            "hashes": [
                "sha256:000546ad01e6389e98626c1367be58efa613fa82a1be98b0c6fc24b563acc6d0",
                "sha256:0d48200bc50cbf498716712129eef819b1729339e34c3ae71656964dac907c28",
                "sha256:23d12eacffa9d0f290c0fe0c4e81ba6d5f3a5b7ac3c30a5eaf0126bf4deda5c8",
                "sha256:37c9ba82bd82eb6a23c2e5acc03055c0e45697253b2393c9a50cef76a3985304",
                "sha256:51503524dd6f152ab4ad1fbd168fc6c30b5795e8c70be4410a64940b3abb55c0",
                "sha256:8041e2de00e745c0e05a502d6e6db310db7faa7c979b3a5877123548a4c0b214",
                "sha256:81fcd96a275209ef117e9ec91f75c731fa18dcfd9ffaa1c0adbdaa3616a86043",
                "sha256:853da4f9563d982e4121fed8c92eea1a4594a2299037b3034c3c898cb8e933d6",
                "sha256:8b4572c334593d449113f9dc8d19b93b7b271bdbe90ba7509eb178923327b625",
                "sha256:9416443e219356e3c31f1f918a91badf2e37acf297e2fa13d24d1cc2380f8fbc",
                "sha256:9854f612e1b59ec66804931df5add3b2d5ef0067748ea29dc60f0efdcda9a638",
                "sha256:99a26afdb82ea83a265137a398f570402aa1f2b5dfb4ac3300c026931817b163",
                "sha256:a19bf883b3384957e4a4a13e6bd1ae3d85ae87f4beb5957e35b0be287f12f4e4",
                "sha256:a9f145660588187ff835c55a7d2ddf6abfc570c2651c276d3d4be8a2766db490",
                "sha256:ac57fcdcfb0b73bb3203b58a14501abb7e5ff9ea5e2edfa06bb03035f0cff248",
                "sha256:bcb530089ff24f6458a81ac3fa699e8c00194208a724b644ecc68422e1111939",
                "sha256:beeabe25c3b704f7d56b573f7d2ff88fc99f0138e43480cecdfcaa3b87fe4f87",
                "sha256:d634a7ea1fc3380ff96f9e44d8d22f38418c1c381d5fac680b272d7d90883720",
                "sha256:d97b0661e1aead761f0ded3b769044bb00ed5d33e1ec865e891a8b128bf7c656"
            ],
            "version": "==0.4.15"
        },
        "gunicorn": {
            "extras": [
                "eventlet"
            ],
            "hashes": [
                "sha256:aa8e0b40b4157b36a5df5e599f45c9c76d6af43845ba3b3b0efe2c70473c2471",
                "sha256:fa2662097c66f920f53f70621c6c58ca4a3c4d3434205e608e121b5b3b71f4f3"
            ],
            "index": "pypi",
            "version": "==19.9.0"
        },
        "monotonic": {
            "hashes": [
                "sha256:23953d55076df038541e648a53676fb24980f7a1be290cdda21300b3bc21dfb0",
                "sha256:552a91f381532e33cbd07c6a2655a21908088962bb8fa7239ecbcc6ad1140cc7"
            ],
            "version": "==1.5"
        },
        "six": {
            "hashes": [
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9",
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb"
            ],
            "version": "==1.11.0"
        }
    },
    "develop": {}
}

[simonschmidt]

Even worse, even if another package explicitly depends on it, it will not be included.

In this example we have:

• celery[redis] Extra dependency on redis
• django-redis Explicit dependency on redis

But we end up with:

pipenv install django-redis 'celery[redis]'
pipenv run python -c 'import redis'  # no error

$ pipenv graph
celery==4.2.1
  - billiard [required: >=3.5.0.2,<3.6.0, installed: 3.5.0.4]
  - kombu [required: >=4.2.0,<5.0, installed: 4.2.1]
    - amqp [required: >=2.1.4,<3.0, installed: 2.3.2]
      - vine [required: >=1.1.3, installed: 1.1.4]
  - pytz [required: >dev, installed: 2018.5]
django-redis==4.9.0
  - Django [required: >=1.11, installed: 2.1.2]
    - pytz [required: Any, installed: 2018.5]
  - redis [required: >=2.10.0, installed: 2.10.6]

$ rm -rf "$(pipenv --venv)"
$ pipenv install
...
$ pipenv run python -c 'import redis' 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'redis'

$ pipenv graph
celery==4.2.1
  - billiard [required: >=3.5.0.2,<3.6.0, installed: 3.5.0.4]
  - kombu [required: >=4.2.0,<5.0, installed: 4.2.1]
    - amqp [required: >=2.1.4,<3.0, installed: 2.3.2]
      - vine [required: >=1.1.3, installed: 1.1.4]
  - pytz [required: >dev, installed: 2018.5]
django-redis==4.9.0
  - Django [required: >=1.11, installed: 2.1.2]
    - pytz [required: Any, installed: 2018.5]
  - redis [required: >=2.10.0, installed: ?]

[techalchemy]

This is a known issue, please keep discussion in the open issue on the topic: #3026

See also: #3039, #3043