setuptools, pip and wheel always in latest version?

[dvzrv]

With current pipenv (2018.5.18, installed with brew, but same behavior also observed on Windows), I'm unable to version-pin setuptools, pip or wheel. However, the circumstances require it to be version pinned, as other tools (twine) will otherwise try to upload a metainfo version they can not cope with to an artifactory instance. Installing setuptools and wheel in a specific version globally/per-user fixes this, but that's also not a solution for our build environment.

pipenv --version
pipenv, version 11.0.2

(btw. the version scheme is still wrong in the --version output! :-)).

Why are these packages excluded from version pinning and can and/or will this be changed?

---

Expected result

Be able to install setuptools, wheel and pip in pinned versions.

Actual result

setuptools, pip, wheel get installed in their latest version, no matter what.

Steps to replicate

Pipfile

[dev-packages]
setuptools  = "<=38.5.2"
wheel       = "0.30.0"
pytest      = "*"
pytest-mock = "*"
pytest-cov  = "*"
coverage    = "*"
mock        = "*"

[packages]
gitpython = "*"
pyyaml = "*"
appdirs = "*"

Pipfile.lock

{
    "_meta": {
        "hash": {
            "sha256": "7fa7214c22ccb6a7e1832afa5228734452bccea95d8c30bedb51cead085e95a6"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "name": "pypi",
                "url": "http://10.1.11.198/artifactory/api/pypi/ni-python-all/simple",
                "verify_ssl": false
            }
        ]
    },
    "default": {
        "appdirs": {
            "hashes": [
                "sha256:9e5896d1372858f8dd3344faf4e5014d21849c756c8d5701f78f8a103b372d92",
                "sha256:d8b24664561d0d34ddfaec54636d502d7cea6e29c3eaf68f3df6180863e2166e"
            ],
            "version": "==1.4.3"
        },
        "gitdb2": {
            "hashes": [
                "sha256:b60e29d4533e5e25bb50b7678bbc187c8f6bcff1344b4f293b2ba55c85795f09",
                "sha256:cf9a4b68e8c4da8d42e48728c944ff7af2d8c9db303ac1ab32eac37aa4194b0e"
            ],
            "version": "==2.0.3"
        },
        "gitpython": {
            "hashes": [
                "sha256:1ec4c44846cf76a1e55769560673a97731849c9d05401e035e607495f10db959",
                "sha256:b60b045cf64a321e5b620debb49890099fa6c7be6dfb7fb249027e5d34227301"
            ],
            "version": "==2.1.10"
        },
        "pyyaml": {
            "hashes": [
                "sha256:16b20e970597e051997d90dc2cddc713a2876c47e3d92d59ee198700c5427736",
                "sha256:3262c96a1ca437e7e4763e2843746588a965426550f3797a79fca9c6199c431f",
                "sha256:592766c6303207a20efc445587778322d7f73b161bd994f227adaa341ba212ab",
                "sha256:5ac82e411044fb129bae5cfbeb3ba626acb2af31a8d17d175004b70862a741a7",
                "sha256:827dc04b8fa7d07c44de11fabbc888e627fa8293b695e0f99cb544fdfa1bf0d1",
                "sha256:bc6bced57f826ca7cb5125a10b23fd0f2fff3b7c4701d64c439a300ce665fff8",
                "sha256:c01b880ec30b5a6e6aa67b09a2fe3fb30473008c85cd6a67359a1b15ed6d83a4",
                "sha256:e863072cdf4c72eebf179342c94e6989c67185842d9997960b3e69290b2fa269"
            ],
            "version": "==3.12"
        },
        "smmap2": {
            "hashes": [
                "sha256:b78ee0f1f5772d69ff50b1cbdb01b8c6647a8354f02f23b488cf4b2cfc923956",
                "sha256:c7530db63f15f09f8251094b22091298e82bf6c699a6b8344aaaef3f2e1276c3"
            ],
            "version": "==2.0.3"
        }
    },
    "develop": {
        "atomicwrites": {
            "hashes": [
                "sha256:240831ea22da9ab882b551b31d4225591e5e447a68c5e188db5b89ca1d487585",
                "sha256:a24da68318b08ac9c9c45029f4a10371ab5b20e4226738e150e6e7c571630ae6"
            ],
            "version": "==1.1.5"
        },
        "attrs": {
            "hashes": [
                "sha256:4b90b09eeeb9b88c35bc642cbac057e45a5fd85367b985bd2809c62b7b939265",
                "sha256:e0d0eb91441a3b53dab4d9b743eafc1ac44476296a2053b6ca3af0b139faf87b"
            ],
            "version": "==18.1.0"
        },
        "coverage": {
            "hashes": [
                "sha256:03481e81d558d30d230bc12999e3edffe392d244349a90f4ef9b88425fac74ba",
                "sha256:0b136648de27201056c1869a6c0d4e23f464750fd9a9ba9750b8336a244429ed",
                "sha256:198626739a79b09fa0a2f06e083ffd12eb55449b5f8bfdbeed1df4910b2ca640",
                "sha256:28b2191e7283f4f3568962e373b47ef7f0392993bb6660d079c62bd50fe9d162",
                "sha256:2eb564bbf7816a9d68dd3369a510be3327f1c618d2357fa6b1216994c2e3d508",
                "sha256:337ded681dd2ef9ca04ef5d93cfc87e52e09db2594c296b4a0a3662cb1b41249",
                "sha256:3a2184c6d797a125dca8367878d3b9a178b6fdd05fdc2d35d758c3006a1cd694",
                "sha256:3c79a6f7b95751cdebcd9037e4d06f8d5a9b60e4ed0cd231342aa8ad7124882a",
                "sha256:3d72c20bd105022d29b14a7d628462ebdc61de2f303322c0212a054352f3b287",
                "sha256:3eb42bf89a6be7deb64116dd1cc4b08171734d721e7a7e57ad64cc4ef29ed2f1",
                "sha256:4635a184d0bbe537aa185a34193898eee409332a8ccb27eea36f262566585000",
                "sha256:56e448f051a201c5ebbaa86a5efd0ca90d327204d8b059ab25ad0f35fbfd79f1",
                "sha256:5a13ea7911ff5e1796b6d5e4fbbf6952381a611209b736d48e675c2756f3f74e",
                "sha256:69bf008a06b76619d3c3f3b1983f5145c75a305a0fea513aca094cae5c40a8f5",
                "sha256:6bc583dc18d5979dc0f6cec26a8603129de0304d5ae1f17e57a12834e7235062",
                "sha256:701cd6093d63e6b8ad7009d8a92425428bc4d6e7ab8d75efbb665c806c1d79ba",
                "sha256:7608a3dd5d73cb06c531b8925e0ef8d3de31fed2544a7de6c63960a1e73ea4bc",
                "sha256:76ecd006d1d8f739430ec50cc872889af1f9c1b6b8f48e29941814b09b0fd3cc",
                "sha256:7aa36d2b844a3e4a4b356708d79fd2c260281a7390d678a10b91ca595ddc9e99",
                "sha256:7d3f553904b0c5c016d1dad058a7554c7ac4c91a789fca496e7d8347ad040653",
                "sha256:7e1fe19bd6dce69d9fd159d8e4a80a8f52101380d5d3a4d374b6d3eae0e5de9c",
                "sha256:8c3cb8c35ec4d9506979b4cf90ee9918bc2e49f84189d9bf5c36c0c1119c6558",
                "sha256:9d6dd10d49e01571bf6e147d3b505141ffc093a06756c60b053a859cb2128b1f",
                "sha256:be6cfcd8053d13f5f5eeb284aa8a814220c3da1b0078fa859011c7fffd86dab9",
                "sha256:c1bb572fab8208c400adaf06a8133ac0712179a334c09224fb11393e920abcdd",
                "sha256:de4418dadaa1c01d497e539210cb6baa015965526ff5afc078c57ca69160108d",
                "sha256:e05cb4d9aad6233d67e0541caa7e511fa4047ed7750ec2510d466e806e0255d6",
                "sha256:f3f501f345f24383c0000395b26b726e46758b71393267aeae0bd36f8b3ade80"
            ],
            "version": "==4.5.1"
        },
        "mock": {
            "hashes": [
                "sha256:5ce3c71c5545b472da17b72268978914d0252980348636840bd34a00b5cc96c1",
                "sha256:b158b6df76edd239b8208d481dc46b6afd45a846b7812ff0ce58971cf5bc8bba"
            ],
            "version": "==2.0.0"
        },
        "more-itertools": {
            "hashes": [
                "sha256:2b6b9893337bfd9166bee6a62c2b0c9fe7735dcf85948b387ec8cba30e85d8e8",
                "sha256:6703844a52d3588f951883005efcf555e49566a48afd4db4e965d69b883980d3",
                "sha256:a18d870ef2ffca2b8463c0070ad17b5978056f403fb64e3f15fe62a52db21cc0"
            ],
            "version": "==4.2.0"
        },
        "pbr": {
            "hashes": [
                "sha256:3747c6f017f2dc099986c325239661948f9f5176f6880d9fdef164cb664cd665",
                "sha256:a9c27eb8f0e24e786e544b2dbaedb729c9d8546342b5a6818d8eda098ad4340d"
            ],
            "version": "==4.0.4"
        },
        "pluggy": {
            "hashes": [
                "sha256:7f8ae7f5bdf75671a718d2daf0a64b7885f74510bcd98b1a0bb420eb9a9d0cff",
                "sha256:d345c8fe681115900d6da8d048ba67c25df42973bda370783cd58826442dcd7c",
                "sha256:e160a7fcf25762bb60efc7e171d4497ff1d8d2d75a3d0df7a21b76821ecbf5c5"
            ],
            "version": "==0.6.0"
        },
        "py": {
            "hashes": [
                "sha256:29c9fab495d7528e80ba1e343b958684f4ace687327e6f789a94bf3d1915f881",
                "sha256:983f77f3331356039fdd792e9220b7b8ee1aa6bd2b25f567a963ff1de5a64f6a"
            ],
            "version": "==1.5.3"
        },
        "pytest": {
            "hashes": [
                "sha256:26838b2bc58620e01675485491504c3aa7ee0faf335c37fcd5f8731ca4319591",
                "sha256:32c49a69566aa7c333188149ad48b58ac11a426d5352ea3d8f6ce843f88199cb"
            ],
            "version": "==3.6.1"
        },
        "pytest-cov": {
            "hashes": [
                "sha256:03aa752cf11db41d281ea1d807d954c4eda35cfa1b21d6971966cc041bbf6e2d",
                "sha256:890fe5565400902b0c78b5357004aab1c814115894f4f21370e2433256a3eeec"
            ],
            "version": "==2.5.1"
        },
        "pytest-mock": {
            "hashes": [
                "sha256:53801e621223d34724926a5c98bd90e8e417ce35264365d39d6c896388dcc928",
                "sha256:d89a8209d722b8307b5e351496830d5cc5e192336003a485443ae9adeb7dd4c0"
            ],
            "version": "==1.10.0"
        },
        "six": {
            "hashes": [
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9",
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb"
            ],
            "version": "==1.11.0"
        }
    }
}

pipenv install

pipenv run pip3 list

Package    Version
---------- -------
appdirs    1.4.3
gitdb2     2.0.3
GitPython  2.1.10
pip        10.0.1
PyYAML     3.12
setuptools 39.2.0
smmap2     2.0.3
wheel      0.31.1

[uranusjr]

I don’t think Pipenv did anything here, and I even filed #1884 specifically because it’s not very convenient to upgrade them. IIRC there is some code to explicitly prevent Pipenv from touching them during installation.

Maybe you can try pipenv run pip install setuptools<=38.5.2 and see if pipenv install updates it? Also you may want to check how they are locked in Pipfile.lock.

[dvzrv]

@uranusjr Thanks for the reply! I added the .lock file (must have overread that part, as I only added the Pipfile). As you can see there, setuptools does not get locked. Neither does wheel.

Force installing a different version after the venv got created indeed works as a workaround, but it would be much saner to allow version pinning with pipenv in the Pipfile for these packages, too.

[techalchemy]

Like most packaging tools, pipenv depends upon you having pip and setuptools. As a consequence it also doesn’t allow you to alter which versions of these things you have in your environment using pipenv itself. If pipenv were able to touch these or allowed you to do that, it would be very easy to destroy your environment. I appreciate that in this specific use case of having an outdated artifactory in a distributed system where you don’t want to tell people to run a manual step, it would be more convenient, but in most other cases the behavior as designed is desirable.

[dvzrv]

@techalchemy well, I just require setuptools to be in a different version. This is not the same as it not being around at all.
If I want to set a different version intentionally, this should be possible.

[dvzrv]

@techalchemy also, this is not about setting my global version of setuptools, but the setuptools version, that will be installed into the virtual environment I'm setting up with pipenv.
I don't understand, why the latest version of setuptools and wheel is forced upon every single virtual environment here.
This renders pipenv completely useless in some use-cases and on top makes its use completely ambivalent, as I'm able to install whatever version of tool I want into said virtual environment after creating it anyhow.
What would be the difference to allowing proper versioning of the base tools I require in the virtual environment to begin with, instead of having the user jump through hoops?

A sane default is of course to choose the latest version of those base tools, but if the user specifically defines a version for them, this should not be ignored!

[uranusjr]

I agree we need to protect certain core packages, but I would tend to agree with @dvzrv here. We should assume the user knows what (s)he’s doing if those packages are explicitly listed in Pipfile.

[techalchemy]

Anything that allows you to alter setuptools can break pipenv fundamentally. I see no reason to build in support for our tools to break themselves.

this is not about setting my global version of setuptools, but the setuptools version, that will be installed into the virtual environment[..] I don't understand, why the latest version of setuptools and wheel is forced upon every single virtual environment here.

1. It is absolutely about your global version of setuptools, because
2. Pipenv doesn’t force anything on your virtualenv. The version of setuptools that lands there is the one you have installed globally.

What you’re asking for is for us to build in support for dependency resolution that permits the resolver to return values that could break our own tools. We aren’t going to do that — if everyone internally is pointed at artifactory and they have to use old setuptools versions your company should consider not making the new version available.

[uranusjr]

I guess the best thing we can do here is to explicitly state what we do and why that is the case. We can emit explicit messages during locking (either strict locking failure or a warning message) to tell the user we can’t obey those specifications because Pipenv needs them to work.

[dvzrv]

Pipenv doesn’t force anything on your virtualenv. The version of setuptools that lands there is the one you have installed globally.

I beg to differ:

pip3 list |grep setuptools
setuptools        39.0.1

pipenv run pip3 list |grep setuptools
setuptools 39.2.0

There is quite a clear automatic update happening here.

As to why we need to achieve this, is that artifactory is probably lacking the speed to deal with newer metadata and twine already making use of it, when using current versions of setuptools.

While I agree, that's something on artifactory's side to fix (will start annoying them on monday), it's also awkward to not be able to set a version for the tools you're going to use in the venv and them being force updated to the latest version automatically.

@uranusjr Thanks for opening #2365!

[techalchemy]

@dvzrv the presence of an upgraded package doesn't really mean anything about pipenv, if you install something else that explicitly and strictly pins a setuptools version it will upgrade setuptools in your virtualenv. If you find evidence of pipenv engaging in this behavior please feel free to provide it, but simply showing state doesn't accomplish that.

[dvzrv]

@techalchemy I'm not sure what you're trying to achieve here. I've provided the above Pipfile and Pipfile.lock. If I remove wheel and setuptools from them (which are just silently ignored anyways), and do a fresh pipenv install for that project, while having setuptools == 39.0.1 installed on my system, setuptools will be around in the venv in version 39.2.0.
When looking at pyyaml, gitpython, gitdb2, smmap2, pip and appdirs, I don't see anything introducing version pinning for setuptools to version 39.2.0.
Therefore I'm assuming this is introduced through pipenv (unless there is something I don't see here).

If you find evidence of pipenv engaging in this behavior please feel free to provide it, but simply showing state doesn't accomplish that.

I'm unsure what else to provide to prove my point here. All logs and literally everything I provided is "state". If that's not valid, no bug reports are ever valid...

That being said: It's not constructive to disregard the collection of information by a participant to reproduce an environment, in which things do not work as intended, while at the same time not providing prove, that this is indeed intended or even documented behavior or disregarding concerns as to why a certain behavior should be changed. This only feels condescending.

I would really like to be able to assume, that with pipenv one can create reproducible environments for projects. With the current example I wonder if this can really be achieved if there's no control over certain packages and the behavior is not clearly being defined regarding those I have no control over.

[techalchemy]

You misunderstand. This isn’t something pipenv does. Obviously I cannot prove a negative so you will have to prove that absent your other requirements, in a brand new environment, pipenv is upgrading setuptools. Since you are filing a bug report against pipenv you need to show that there is a bug — I don’t have your system or your environment, and while I appreciate that this is frustrating, it is not reasonable to ask us to chase down issues which we can’t reproduce and for which the only evidence is that after doing many things, an undesirable state is attained.

Surely you can understand why I am pushing back rather hard. The tenor of this issue has been that an outcome occurred and absent any other lines of inquiry and with resistance to the idea that there could possibly be any other cause, the conclusion you came to is that we are responsible for investigating and solving this problem — and that is a bit antagonistic. I get your frustration. However, we are a small team of volunteers with limited resources. If you find a bug you are welcome to report it, but unless you provide sufficient information to reproduce it and evidence that it is a problem in pipenv, we probably aren’t going to spend any resources on it. If the bug is important to you, we expect that you will at least assist us in determining whether it exists rather than assuming it does.

This project is a collaborative open source one — basically the process works better if you collaborate instead of blame.

To your last point. I already described the behavior. Go ahead and create a fresh environment by just running pipenv install six. I don’t really appreciate the passive aggressive questioning, I’d be happy to work with you if there is a bug. You need to demonstrate that by providing information about your system. If you prefer the alternative i will simply point you at the code of conduct.