Lock file differs depending on Python version of Pipenv installation

[mpolden]

Pipenv seems to create a different lock file depending on whether Pipenv was installed with pip2 or pip3.

Example with pipenv installed using pip2 install --user pipenv:

$ which pipenv
/Users/martin/Library/Python/2.7/bin/pipenv

$ pipenv install apscheduler
Creating a virtualenv for this project…
⠋Using base prefix '/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/martin/.local/share/virtualenvs/foo-mzLZXOua/bin/python3.6
Also creating executable in /Users/martin/.local/share/virtualenvs/foo-mzLZXOua/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/martin/.local/share/virtualenvs/foo-mzLZXOua
Creating a Pipfile for this project…
Installing apscheduler…
Collecting apscheduler
  Using cached APScheduler-3.3.1-py2.py3-none-any.whl
Collecting tzlocal>=1.2 (from apscheduler)
Collecting pytz (from apscheduler)
  Using cached pytz-2017.2-py2.py3-none-any.whl
Requirement already satisfied: setuptools>=0.7 in /Users/martin/.local/share/virtualenvs/foo-mzLZXOua/lib/python3.6/site-packages (from apscheduler)
Collecting six>=1.4.0 (from apscheduler)
  Using cached six-1.11.0-py2.py3-none-any.whl
Installing collected packages: pytz, tzlocal, six, apscheduler
Successfully installed apscheduler-3.3.1 pytz-2017.2 six-1.11.0 tzlocal-1.4

Adding apscheduler to Pipfile's [packages]…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (019ead)!

$ cat Pipfile
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]
apscheduler = "*"

$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "10b2336ecdb1ebf47f350b4b23ada696f0662969a53d4c798a51bf073c019ead"
        },
        "host-environment-markers": {
            "implementation_name": "cpython",
            "implementation_version": "3.6.3",
            "os_name": "posix",
            "platform_machine": "x86_64",
            "platform_python_implementation": "CPython",
            "platform_release": "17.0.0",
            "platform_system": "Darwin",
            "platform_version": "Darwin Kernel Version 17.0.0: Thu Aug 24 21:48:19 PDT 2017; root:xnu-4570.1.46~2/RELEASE_X86_64",
            "python_full_version": "3.6.3",
            "python_version": "3.6",
            "sys_platform": "darwin"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "apscheduler": {
            "hashes": [
                "sha256:bc9f96e498adb362beb5f1d3715a2570d100183add4ace5227c1a7d5dbaac900",
                "sha256:f68874dff1bdffcc6ce3adb7840c1e4d162c609a3e3f831351df30b75732767b"
            ],
            "version": "==3.3.1"
        },
        "funcsigs": {
            "hashes": [
                "sha256:330cc27ccbf7f1e992e69fef78261dc7c6569012cf397db8d3de0234e6c937ca",
                "sha256:a7bb0f2cf3a3fd1ab2732cb49eba4252c2af4240442415b4abce3b87022a8f50"
            ],
            "markers": "python_version == '2.7'",
            "version": "==1.0.2"
        },
        "futures": {
            "hashes": [
                "sha256:c4884a65654a7c45435063e14ae85280eb1f111d94e542396717ba9828c4337f",
                "sha256:51ecb45f0add83c806c68e4b06106f90db260585b25ef2abfcda0bd95c0132fd"
            ],
            "markers": "python_version == '2.7'",
            "version": "==3.1.1"
        },
        "pytz": {
            "hashes": [
                "sha256:c883c2d6670042c7bc1688645cac73dd2b03193d1f7a6847b6154e96890be06d",
                "sha256:03c9962afe00e503e2d96abab4e8998a0f84d4230fa57afe1e0528473698cdd9",
                "sha256:487e7d50710661116325747a9cd1744d3323f8e49748e287bc9e659060ec6bf9",
                "sha256:43f52d4c6a0be301d53ebd867de05e2926c35728b3260157d274635a0a947f1c",
                "sha256:d1d6729c85acea5423671382868627129432fba9a89ecbb248d8d1c7a9f01c67",
                "sha256:54a935085f7bf101f86b2aff75bd9672b435f51c3339db2ff616e66845f2b8f9",
                "sha256:39504670abb5dae77f56f8eb63823937ce727d7cdd0088e6909e6dcac0f89043",
                "sha256:ddc93b6d41cfb81266a27d23a79e13805d4a5521032b512643af8729041a81b4",
                "sha256:f5c056e8f62d45ba8215e5cb8f50dfccb198b4b9fbea8500674f3443e4689589"
            ],
            "version": "==2017.2"
        },
        "six": {
            "hashes": [
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb",
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9"
            ],
            "version": "==1.11.0"
        },
        "tzlocal": {
            "hashes": [
                "sha256:05a2908f7fb1ba8843f03b2360d6ad314dbf2bce4644feb702ccd38527e13059"
            ],
            "version": "==1.4"
        }
    },
    "develop": {}
}

And pipenv installed with pip3 install --user pipenv:

$ which pipenv
/Users/martin/Library/Python/3.6/bin/pipenv

$ pipenv update
Updating all dependencies from Pipfile…
Found 3 installed package(s), purging…
Uninstalling APScheduler-3.3.1:
  Successfully uninstalled APScheduler-3.3.1
Uninstalling pytz-2017.2:
  Successfully uninstalled pytz-2017.2
Uninstalling tzlocal-1.4:
  Successfully uninstalled tzlocal-1.4

Environment now purged and fresh!
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (019ead)!
Installing dependencies from Pipfile.lock (019ead)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 4/4 — 00:00:01
To activate this project's virtualenv, run the following:
 $ pipenv shell
All dependencies are now up-to-date!

$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "10b2336ecdb1ebf47f350b4b23ada696f0662969a53d4c798a51bf073c019ead"
        },
        "host-environment-markers": {
            "implementation_name": "cpython",
            "implementation_version": "0",
            "os_name": "posix",
            "platform_machine": "x86_64",
            "platform_python_implementation": "CPython",
            "platform_release": "17.0.0",
            "platform_system": "Darwin",
            "platform_version": "Darwin Kernel Version 17.0.0: Thu Aug 24 21:48:19 PDT 2017; root:xnu-4570.1.46~2/RELEASE_X86_64",
            "python_full_version": "2.7.14",
            "python_version": "2.7",
            "sys_platform": "darwin"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "apscheduler": {
            "hashes": [
                "sha256:bc9f96e498adb362beb5f1d3715a2570d100183add4ace5227c1a7d5dbaac900",
                "sha256:f68874dff1bdffcc6ce3adb7840c1e4d162c609a3e3f831351df30b75732767b"
            ],
            "version": "==3.3.1"
        },
        "pytz": {
            "hashes": [
                "sha256:c883c2d6670042c7bc1688645cac73dd2b03193d1f7a6847b6154e96890be06d",
                "sha256:03c9962afe00e503e2d96abab4e8998a0f84d4230fa57afe1e0528473698cdd9",
                "sha256:487e7d50710661116325747a9cd1744d3323f8e49748e287bc9e659060ec6bf9",
                "sha256:43f52d4c6a0be301d53ebd867de05e2926c35728b3260157d274635a0a947f1c",
                "sha256:d1d6729c85acea5423671382868627129432fba9a89ecbb248d8d1c7a9f01c67",
                "sha256:54a935085f7bf101f86b2aff75bd9672b435f51c3339db2ff616e66845f2b8f9",
                "sha256:39504670abb5dae77f56f8eb63823937ce727d7cdd0088e6909e6dcac0f89043",
                "sha256:ddc93b6d41cfb81266a27d23a79e13805d4a5521032b512643af8729041a81b4",
                "sha256:f5c056e8f62d45ba8215e5cb8f50dfccb198b4b9fbea8500674f3443e4689589"
            ],
            "version": "==2017.2"
        },
        "six": {
            "hashes": [
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb",
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9"
            ],
            "version": "==1.11.0"
        },
        "tzlocal": {
            "hashes": [
                "sha256:05a2908f7fb1ba8843f03b2360d6ad314dbf2bce4644feb702ccd38527e13059"
            ],
            "version": "==1.4"
        }
    },
    "develop": {}
}

Using pipenv --two update in the last example gives the same result.

This effectively means that it's not possible to use a Python 3 Pipenv to generate a Pipfile.lock or requirements.txt that is usable in a Python 2 environment.

[kennethreitz]

Are you using the latest version?

[mpolden]

Yes:

$ pipenv --version
pipenv, version 8.2.6

[mpolden]

This may be related to #848. The Python 3 installation of pipenv seems to ignore the environment markers in the Pipfile created with the Python 2 installation.

[vphilippon]

Even if pipenv was to respect environment markers, the Pipfile.lock would not be guaranteed to be unchanged between Python 2 and Python 3 (nor across different OS).

To list and resolve the dependencies, we fetch the wheels of the packages. These wheels can be different, and provide different dependencies, depending on whether we are on Python 2 or Python 3 (ex: foo-1.0.0-py2-none-any.whl vs foo-1.0.0-py3-none-any.whl). This check and selection is done by pip.

To make/compile a "universal" Pipfile.lock (or requirements.txt), we'd need to respect environment markers, and patch pip to iterate through every wheel/sdist available for a given package version and compute the full set of dependencies for each (as each might yield different dependencies), tagging each of those with the appropriate environment markers. This is not a one-line-fix 😄.

We've hit the same issue with pip-tools, which pipenv uses for dependency resolving (with a few patches).

[mpolden]

Thanks, I understand the perfect solution might be complicated.

What surprised me was that pipenv lock --two does not do what I expect either (create a Python 2-compatible lock file).

Maybe lock could use the Python 2 interpreter to generate the lock file when --two is given? Or do you think that is just as complicated to solve?

[vphilippon]

From the Readme:
--three / --two Use Python 3/2 when creating virtualenv.
I would assume these options only apply when first creating the virtualenv, which would likely only be done once, but that goes beyond my knowledge of pipenv and what's the intended use for these options. You'll need the input of someone else here.

[erinxocon]

Closing issue! I think the docs make it clear what --two and --three do.

[mpolden]

The issue is not resolved though. The documentation should at the very least state that to create Python 2 compatible Pipfile.lock or requirements.txt requires a Python 2 installation of Pipenv.

[erinxocon]

@mpolden I think that's pretty self explanatory. I'm open to feedback from some other maintainers though! @vphilippon @nateprewitt @techalchemy

[techalchemy]

Saw this the other day but wasn’t sure what to say... how could something you installed using python 3 and therefore runs in python 3 then run itself in the python 2 interpreter? I mean, there are conceivably ways but it’s a horrible practice and it doesn’t work very well... why would you want this when you can do a python2 install of pipenv?

[mpolden]

why would you want this when you can do a python2 install of pipenv?

In my case I think didn't think much about it. I use a Python 3 environment in my dev environment, but deploy to a Python 2 environment.

There's also the options --two and --three that states:

Use Python 3/2 when creating virtualenv

which I assumed would provide the needed compatibility.

Additionally, the docs seems to nudge the user towards Python 3. From https://docs.pipenv.org/install.html#make-sure-you-ve-got-python-pip:

This guide is written for Python 3, however, these instructions should work fine on Python 2.7—if you are still using it, for some reason.

And finally, deterministic builds seems to be a key selling point in the features list. From https://docs.pipenv.org/#pipenv-features

Enables truly deterministic builds, while easily specifying only what you want.
Generates and checks file hashes for locked dependencies.

It was not obvious to me that the results of dependency resolution would depend on the Python version used by pipenv itself (and not the virtualenv) , and I couldn't find any mention of this behavior in the documentation.

[vphilippon]

It was not obvious to me that the results of dependency resolution would depend on the Python version used by pipenv itself (and not the virtualenv) , and I couldn't find any mention of this behavior in the documentation.

Well, my thought on this is that if pipenv is going to offer to create/manage virtualenvs that can be either Python 2 or Python 3, then the dependency resolution and locking for that environment should respect the Python version of that environment, of course.
If the dependency resolution is based on the root version of python used to install pipenv itself, I'd call it broken, and users will get into trouble for sure.
"You can create Python 2 or Python 3 venv, but the locking will be valid if your venv Python version matches the root python version used to install pipenv to begin with." Oh hell no 😨 .

I actually thought this was dealt with somehow with a pip-tools and pip patch done by Kenneth. I haven't looked deeply into that though yet. If it's effectively not dealt with, this really needs to be addressed (and it sounds like I'll have to find some free time to actually go and look into that 😄 ).

[nateprewitt]

Yeah, there's something weird going on here. Which version of Python you install pipenv on should not be affecting the lock file. A venv established with --two vs --three will have differing results and the os can affect the final output, but the system Python version is supposed to be irrelevant.

[erinxocon]

So if I'm understanding correctly: If Python3 is the system python and you create a virtualenv using python2 from a pipenv that's installed with py3, the lock file will be different than if pipenv was installed with a system python of 2.7? If so what are the changes between the lock file?

[vphilippon]

@erinxocon You're correct. The difference would be that the lockfile would contain Python 3 dependencies instead of Python 2 dependencies.

In short: the dependencies would be computed based on the python environment that pipenv is in itself (likely, the root env outside of any venv), instead of the environment for which its supposed to comput the dependencies.

But, it might be a bit more complex than that. Python version specific dependencies can be determined from various means. It can be with environment markers, or by the py2 and py3 wheels listing different dependencies, or some specific code in the setup.py when dealing with sdists. At this point, I cannot tell if one or all of the above are impacted by the "root python version".

The whole "get stuff for the right environnement" is dealt by pip. It all boils down to what pip thinks the environment is for each of those variant.

[techalchemy]

@vphilippon I know he was working on this around the time of this commit: 7d2927f -- might be worth looking around that time, i can dig into this in a little bit

[techalchemy]

@vphilippon @erinxocon here is the list of commits I can find with regard to locking / patching pip / python version hacks / dependency resolution:

• 9c12ba5
• 234931b
• 7010383
• e4cba86
• ae4591b
• 4c000e4
• d69d116

Not sure what to make of all of this but the answer to this issue lies somewhere in all of these commits, I would guess.

[tuukkamustonen]

Similar discussion in jazzband/pip-tools#563 (especially this, this and this comment).

If it's not possible, or feasible, to support "universal" lock files (which is just ok), it should be at least clearly documented, so user doesn't get false expectations. You simply get the lock file for the python version / OS platform you compiled on (until future improvements may or may be introduced)?

(But I assume this is offtopic, as the original issue was about using virtualenv python environment and not pipenv installation environment to calculate the lock file.)

[erinxocon]

So I've been thinking about this @vphilippon @techalchemy and @nateprewitt. Is this related to #1014 with the wrong python version being used to create a lock file?

[techalchemy]

@erinxocon yes, for sure. It's the same issue. I know @vphilippon is looking at this some when he can, I've spent a few spare minutes on this when I have them but it really requires a deep dive and a lot of focused effort to figure out where the last bit of trickery needs to happen.

[vphilippon]

Small update: This is in my mind (days and night 😄 ), and I hope to be able to dive into that, but this won't be a mere 10 minutes task. I have to find the time to dive into this, so sorry for the delay.

[mpolden]

@vphilippon Good to hear you're working on it. Thanks!

[ghost]

In our workflow, we use macOS for development & docker for deployment to Linux (AWS Lambda or EC2 instances). Our codebase is a mix of Python2 & Python 3 services deployed independently. While investigating Pipenv, I realized that the Pipfile.lock generated on macOS for a python2 project fails to work within docker. I have installed pipenv in python3 in macOS. The culprit is cryptography package which includes ipaddress packed on python < 3. The fix for this issue will provide the solution for this use case.

However, lockfile being dependant on the OS is contrary to my expectations set up by npm, go dep, nuget & other tools. If there are no plans to make lockfile independent of OS, this point should be clearly mentioned in the documentation.

[techalchemy]

@surajbarkale-dolby lockfiles aren't OS dependent, that metadata is included just for informative purposes

[ncoghlan]

Restating the core of the problem: we really want the lockfile to always capture and pin a package superset that covers both platform independent dependencies, and potential platform dependent dependencies (those that include environment markers). This would then be more akin to what we do for unpinned dependencies in wheel files and setup.py files: all the dependencies for every platform would always be listed, but exactly which ones would get installed would be based on the listed environment markers.

So while "run pipenv with the same Python you're going to deploy with" is OK as an interim workaround (albeit one that limits pipenv's suitability for managing library test suites), I'm thinking a long term solution is going to look more like the following:

1. Assume all environment markers are "true" when generating the lock file (so everything gets listed and pinned, even items that aren't needed on the platforms generating the lock file).
2. If a particular package is never brought in unconditionally, then add a suitable "markers" entry (ala https://github.com/pypa/pipfile/blob/master/examples/Pipfile.lock#L268), with different markers from different dependency declarations being or'ed together as necessary.
3. Evaluate the markers normally when installing from the lock file

There are still ways for such an approach to fail (e.g. platform specific dependencies that don't even support having their dependency metadata extracted on other platforms), but I think it's the closest we're going to be able to get to the objective of having the dependencies listed in the lock file be platform independent.

[techalchemy]

@ncoghlan I agree wholeheartedly, and I actually think we can accomplish this. Previously we approached this by forcing the resolver to use a fake sys.version_info tuple, but that caused some errors downstream, particularly because of areas where pip makes assumptions about strings depending upon the python version and it derives that information from sys,version_info. My first attempt to work around this involved overriding the __getitem__ method of our namedtuple to allow for exceptions, which is kind of ugly: https://github.com/pypa/pipenv/pull/1082/files

However, pip only makes this assumption in one place, and since we are vendoring a patched version of pip, I believe we can just replace the conditional with some exception handling anyway.

Assume all environment markers are "true" when generating the lock file (so everything gets listed and pinned, even items that aren't needed on the platforms generating the lock file).

Unfortunately this is the one I'm not so sure about. All of the resolver tools in pip depend on ABI flags, so essentially we would have to tell the resolver to resolve against all possible ABI flag combinations, and not even just for the current platform. How far does this go? Do we include different pythons as well? I haven't tested this quite yet, but I have a general sense that this will not be straightforward because of:

(e.g. platform specific dependencies that don't even support having their dependency metadata extracted on other platforms)

In particular I am thinking of pywin32 as it is now named (with a shim redirecting pypiwin32) which simply imports win32 apis in its setup.py -- if we generated and resolved against all platform ABI combos this would just throw some random ImportError, or bubble up as something else, on platforms other than windows. I don't know how often this occurs in terms of how many packages do this type of thing, but many users do require conditional installation of this library on windows.

Interestingly, PyPI json just began including platform, requires_dist, and requires_python keys but they are not reliably populated. If they do become populated, this may provide one path when dealing with the official repository anyway.

[uranusjr]

In the case of sys.version_info patching, is it plausible to actually run the resolver against the virtualenv Python? I feel it might work to run the resolution in a subprocess inside the virtualenv, but maybe there’re unsurmountable difficulties I’m not aware of.

[techalchemy]

@uranusjr in some issue or other I recently proposed that, but in a recent discussion with @kennethreitz he reminded me that we did try that without much success.

I did however have some success passing information back and forth across a subprocess call to extract ABI tags and such because you can actually pass that information to the base pip resolver (on which pip-tools relies), but it was insufficient for actual resolution because pip-tools creates InstallRequirements which must respect the python version.

This has become quite tangled, and on several occasions I have contemplated isolating and reimplementing the resolver functionality in pipenv so that it works the way we want it to, rather than performing all of these quite frankly ugly hacks to attempt to force existing resolvers to cooperate. Now that I've actually put that in writing, I'd be curious to hear your thoughts. /cc @ncoghlan @nateprewitt @erinxocon @vphilippon

[ncoghlan]

@techalchemy Yeah, I think this is a case where attempting to re-use the existing installation-focused resolvers is actually the root of the current problem, since the requirements for platform-independent lock file resolution are just different enough that we've ended up trying to fit a square peg into a hole with slightly rounded corners. If Python didn't support environment markers I believe the existing approach would work, but as things stand, the platform-dependent nature of dependency installation is leaking out into the lockfile generation.

For cases like pywin32, where setup.py dist_info fails on non-Windows systems, I think the best we're going to be able to do is:

• check PyPI for transitive dependency info, and use it if available
• if PyPI doesn't have anything, then fail the lock operation by default (but have an opt-in toggle like --skip-incompatible that converts any such errors to warnings instead)

And we'd then advise publishers to either 1) make their metadata extraction work cross-platform; or 2) upload wheel files so PyPI can extract and publish the metadata via the JSON API

@uranusjr I originally also thought running the lock file generation inside the venv would be enough, but then I later realised it still isn't enough to solve the problem:

• if you abide by environment markers, but use a Py3 venv to generate the lock file, you'll still miss locking Py2-only dependencies (i.e. running in the venv just inverts the current "lockfile was generated with the wrong Python version" problem, it doesn't fix it)
• generating the lockfile on Linux or Mac OS X would still miss Windows-specific dependencies and vice-versa

[ncoghlan]

@techalchemy That said, I think the problem we're facing here might actually be relevant to pip-compile as well, it's just that because pip-tools doesn't try to enforce a 1-to-1 correspondence between input files and lock files, it's easier to work around the issue by generating separate requirements files for different target environments.

[ncoghlan]

I think @joshfriend's comment at #1255 (comment) highlights another case where the current platform dependence of lock files is problematic: if you destroy the virtual environment and recreate it with a different Python version or a different Python implementation, pipenv won't regenerate it.

That's actually the desired behaviour (since the lock file is nominally platform independent, change versions or implementations shouldn't inherently lead to different lock file contents outside the host-environment-markers section - only changing Pipfile should do that), but it's currently problematic due to the eager evaluation of environment markers at lock file generation time.

[ncoghlan]

Chatting to @kennethreitz this evening, he's interested in trying the "Resolve dependencies using the virtualenv Python" approach again, and amending pipenv lock to always set the requires_python field in Pipfile.lock even if it isn't set in Pipfile.

Implicitly setting requires_python based on the target version used to generate the lock file would make the current version dependence of the format explicit (and 4566e4d has already updated the docs to be more explicit on that front).

While I think this would be an improvement over the status quo for the primary deployment use case, I think it may prove problematic for the test suite management use case: https://docs.pipenv.org/advanced/#testing-projects

[ncoghlan]

As of v11, managing Py2 environments with Py3 (and vice-versa) should be markedly improved, as ecf7842 means the dependency resolution for lock file generation now runs in the context of the venv Python.

In particular, if a setuptools extension runs a subprocess (or otherwise bypasses pipenv's attempt to override the nominal Python version), that subprocess will also run in the virtual environment, and hence see the correct version of Python (and other environmental details).