pipenv lock --keep-outdated does not respect Pipfile.lock constraints

[wasinski]

Issue description

When I have Pipfile with a non-locked dependency (*) but the dependency is locked in Pipfile.lock
and then I lock dependencies with --keep-outdated flag in the current constraints latest available version of the package is used.

Pipfile

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
pytest = "*"

Pipfile.lock

[...]
        "atomicwrites": {
            "hashes": [
                "sha256:03472c30eb2c5d1ba9227e4c2ca66ab8287fbfbbda3888aa93dc2e28fc6811b4",
                "sha256:75a9445bac02d8d058d5e1fe689654ba5a6556a1dfd8ce6ec55a0ed79866cfa6"
            ],
            "version": "==1.3.0"
        }
        "pytest": {
            "hashes": [
                "sha256:27abc3fef618a01bebb1f0d6d303d2816a99aa87a5968ebc32fe971be91eb1e6",
                "sha256:58cee9e09242937e136dbb3dab466116ba20d6b7828c7620f23947f37eb4dae4"
            ],
            "index": "pypi",
            "version": "==5.2.2"
        }
[...]

verbose output

Current constraints:
  pytest

Expected result

verbose output

Current constraints:
  pytest==5.2.2

In the case of pytest it results in not installing proper sub-dependencies. Version 5.3 dropped the requirement for atomicwrites, so the newly generated Pipfile.lock does not contain it, but since version of pytest which I actually use is 5.2.2 I'm getting an ImportError

[mcallaghan-bsm]

Same as #3975?

[wasinski]

yes, this is the same issue, sub-dependencies are updated due to the behavior described above.

[januszm]

I'll link this one too: #4055

[grazius]

Nothing new on this issue ?

It's a real problem to don't have the ability to generate requirements that stricly respect Pipfile.lock
pipenv lock --keep-outdated --requirements

Maybe only the integration of the --ignore-pipfile flag can allow this functionnality to work well ?

[matteius]

Can this be rechecked on the latest pipenv==2022.11.5?

[matteius]

I am reposting this explanation from a similar issue report recently #4055 (comment):

Sorry, I re-read the issue report, and I have to explain that pipenv install package_name has to relock -- this is part of the dependency resolution to be able install package_name and respect the other dependencies. Say that you have pinned to an old version of Django and package_name requires a new version of Django, nobody would want the side effect that pipenv install violates its own constraints, hence it has to go through the dependency resolution phase.

But it should never touch other packages in Pipfile.lock unless they are direct dependencies of the newly installed package.

This isn't true, what you are asking for is that the default behavior of install to be pipenv lock --keep-outdated but that is a flawed implementation and we should not prefer users to use that as the default. Furthermore, it seems you can already pass --keep-outdated to pipenv install, which sounds like what you want.

What you should be doing instead is actually picking proper specifiers for your project in the Pipfile such that you are not inconvenienced by re-locking when installing new packages, then you won't get bugged outcomes that are possible from using --keep-outdated in the first place.

Thanks for pointing out that other comment in that thread that has so many likes -- to me it represents a complete misunderstanding of what the pip resolver is capable of and the kind of guarantees it provides when you use it properly. Either way --keep-outdated or not, we invoke the pip resolver and in the case of --keep-outdated we only update some of the results which can lead to lock files that are not installable or will have unexpected compatibility outcomes.

[matteius]

The issue itself is a strong indicator that --keep-outdated should be removed from pipenv. I mean the whole premise of it is that if you already have a dependency in your lock file it won't update it. Why even use it?

[januszm]

@matteius apologies for splitting your comment in parts, but it'll be easier to reply to a couple of statements you made this way:

it should never touch other packages in Pipfile.lock unless they are direct dependencies of the newly installed package.

This isn't true,

I believe this is your private opinion, but as I'm coming from 'bundler' world in Ruby I can confirm that this at least is true there, to back it up with something, I'll at least mention the "principle of least surprise", which pipenv is currently violating. Also the fact that pipenv install alters other, unrelated package dependencies sounds like "feature envy".

This also resembles the "git commit" situation. Compare git add . or git add -A with precisely selecting files and changes. It seems to me that it is generally accepted that a commit should contain an "atomic" change in the system, and I would treat package dependencies management similarly. pip install packageA with an accidental upgrade of unrelated packageB is like adding "all" changes in working directory to a specific commit in git.

What you should be doing instead is actually picking proper specifiers for your project in the Pipfile such that you are not inconvenienced by re-locking when installing new packages

Most people do that already, but still, even if I specify pymssql = "~=2.2" and thenpipenv install numpy it may upgrade pymssql , which I then have to test separately and what's more, if by any chance pymssql minor upgrade introduces a "bug", which may always happen, it will make debugging harder (it could be avoided in the first place).

pipenv lock --keep-outdated but that is a flawed implementation and we should not prefer users to use that as the default

Lastly , this is a different story. A flawed implementation does not mean the whole concept is wrong. At least pipenv should have a sane option of keeping the unrelated packages untouched. Perhaps fixing this implementation will be a good compromise.

[matteius]

Also the fact that pipenv install alters other, unrelated package dependencies sounds like "feature envy".

The other unrelated packages are actually related to your Pipfile and the specifiers you install there. All dependencies may affect the resolution of the other dependencies and so a complete resolver phase has to happen and this is not just my opinion, without this, you get the kind of behavior that --keep-outdated provides which leads to plenty of issue reports because the resolver results were not respected. The only input into the resolver is the Pipfile and its the end user's responsibility to pick reasonable specifiers for their top level packages and things they care to pin.

even if I specify pymssql = "~=2.2" and thenpipenv install numpy it may upgrade pymssql , which I then have to test separately and what's more, if by any chance pymssql minor upgrade introduces a "bug", which may always happen, it will make debugging harder (it could be avoided in the first place).

it could be avoided in the first place by recognizing this fact (which you have) and then pinning it in your Pipfile until you are ready to put the time into testing the upgrade of pymssql.

A flawed implementation does not mean the whole concept is wrong. At least pipenv should have a sane option of keeping the unrelated packages untouched.

This is not how the pip resolver needs to work. you think the packages are unrelated, but they are related to the other packages you are resolving/installing and those packages apparently did not restrict it in the way you want, so please use the Pipfile for this purpose.

The common problem theme here with --keep-outdated is that "I wanted pipenv to do what I want it to, but I don't want to tell it how." There is no magic forumla to figure out that you want your pymssql pinned, but that you want all of your dependencies to all resolve and install correctly, so you have to be explicit and not look for a magic bullet like --keep-outdated. We are likely to deprecate this flag in the near future and remove it after that.

[matteius]

@januszm I noticed you thumbs downed my last comment, but its the truth. I re-read your comment, one more advice:

Most people do that already, but still, even if I specify pymssql = "~=2.2"

That is because the tilde operator allows it to upgrade the minor version of the specifier, if you want it pinned to 2.2 use pymssql = "==2.2"

I am happy to discuss this more and further explain the problems of --keep-outdated, but I am not sure what you are objecting to.

[matteius]

I can see where my response was a bit quick the other night, but it was late for me. I'd like to suggest that while I once thought maybe the --keep-outdated implementation can improve, I really don't think that it can at this point a year into studying and working on the pipenv code base. Here is why:

1. We are definitely not interested in making a "bespoke" resolver that is divergent from the resolver pip has. While it is true we are using pip internals to utilize the pip resolver, and have one security related patch for index resolution determinism, in general we are not trying to modify pip or its resolver within pipenv.

Given this, what happens is the lock file is generated from a resolution of all the Pipfile specifiers. When you use a tilde, you are using a loose specifier meaning you don't mind if the package version changes--there is a formula for how the tilde is computed that I won't go into here/now (would require me to look it up again). My focus on this thread is around the resolver phase. Your inputs go in an a resolved set of packages comes out.

Following that, how keep outdated works is it takes the lock resolution of the new resolver result, and merges it with what the older resolver result was, in a simplistic way of saying what ever package are already in the lock file I'll keep. This logic doesn't have any resolver accountability, and so the result may not be installable or worse, it may install but be totally incompatible. There really isn't a way to solve for this without having the specifers you care about in the Pipfile such that they are used in the new lock resolution and respected in the final lock file output. Here is the code I am referring to: https://github.com/pypa/pipenv/blob/main/pipenv/core.py#L1209-L1228

Sorry if I stepped on any toes, but I think its important to reduce the specification of pipenv to what can actually be supported without issues. The docs only reference this about --keep-outdated:

These are pretty old docs too and the statement doesn't make a lot of sense as written. I think what it should be suggesting is to only keep top level dependencies and ones you know you want pinned in the Pipfile and let the lockfile manage the rest. You don't need --keep-outdated for that, in fact it makes it worse.

[januszm]

@matteius

I noticed you thumbs downed my last comment, but its the truth.

I believe this is more of your own opinion rather than universal truth. I noticed that you and probably other pipenv maintainers have a fundamentally different understanding of how a dependency management system should work when it comes to package installation. Since there are so many differences here, it seems that the Python community needs another system as an alternative to pipenv. It's a pity, because it's always more effective to focus all efforts and budget on improving one solution, as was the case with bundler for Ruby. Perhaps poetry will be developed in the direction expected by the other part of the community that has spoken on this and similar issues. E.g. see #3150 (comment) as this comment sums up the expectations of "the other part" of the community.

I think I'll end my participation in this discussion here because neither I nor others present here (and in related issues) will convince you to change the design decisions regarding the installation, nor, as you have probably noticed, you will not convince others.

[matteius]

Well until I hear an actual proposal that would make --keep-outdated workable other than an opinion that it should work without explaining how, then the plan is to deprecate and remove it from the codebase. I have been maintaining pipenv for about 1 year now, and we are open to changing and fixing things, even improving the interface--for which removing --keep-outdated would be a big improvement because it would not create misleading expectations about what is possible.

[sidneydemoraes]

I know I'm way late for the discussion, but I'd like to add a case that might justify the feature request.
We had a project fixing dateparser to version 1.1.0.
It depends on regex, but its setup.py file does not fix the version of regex.
When we went to production we "luckily" got the updated (and broken) version of regex even though we used the --keep-outdated and took literally hours to find out what was going on.
So I would say that something needs to be fixed for sure. Fact is that Pipfile.lock were not respected even though we had fixed our main dependency on Pipfile.

[matteius]

@sidneydemoraes since we removed --keep-outdated there has been a similar feature request, you would probably be better off tracking: #5914

EDIT: Actually it sounds like you are saying that you did not pin regex in your Pipfile, just dateparser, and somehow pipenv should be able to figure out that despite the constraint allowing the latest regex, that it shouldn't use it? This gets back to the fact pipenv cannot magically figure out that a newer version breaks and wasn't restricted properly. The recommendation is to add additional Pins to the Pipfile as that is the spec, and Pipfile.lock is the generated lock file. upgrade provides better control over how pacakges in the lock get affected, but it still wouldn't solve the example you are describing.