Upgrading of packages because keep outdated was removed in latest version of pipenv

[matejsp]

So. The old one of best functionalities was removed from pipenv (keep-outdated). We use it because we selectively update dependencies (transitive). Especially for non dev environment.

Our current workflow was to update pipfile and do sync with keep outdated. It is fine, but I cannot

Issue description

I am trying to get around with (upgrading 1.4.1 to 1.5.1):

PIPENV_IGNORE_VIRTUALENVS=1  pipenv upgrade mypy==1.5.1 --dev

Issues:

• It deletes all whitespaces making pipfile very unreadable and hard to revert all the corruption before comiting.
• And you can no longer manage upgrades by updating pipfile. You always have to remember how to call a command.
• It corrupts pipefile if you don't pass --dev in case of mypy ... You are left with two different versions in lock file (one for non dev, one for dev)
• It does not work if you merge from different branches. Because you must manually go over the pipefile and call update manually and still you won't be sure that you didn't miss something.

Trying with second more promising alternative: #5730

Setting mypy to 1.5.1 in pipfile. And running update. According to comment:

calls upgrade under the hood, upgrades the relevant packages in the lock file and installs them.

I expected to update relevant packages, but it does not, because it update them all. It behaves exactly the same as lock.

Expectation for update functionality was that is would check version diff between pipefile and lock file and in a loop call update for all packages that are in lock file and their pinned version is different in a lock file. But it just does the lock.

Expected result

Steps:

• go over pipfile and lock file
• find all the different versions (if I change mypy to 1.5.1 so it should find just that change)
• call upgrade for mypy only (or others in for loop if multiple changes)
• result is that only mypy would be updated
• it shouldn't mess with old (& broken) keep outdated functionality and still achieve the same thing as before.

Actual result

It changes all the version to latest.
Currently it is quite hard for use to use pipenv.

[matteius]

• It deletes all whitespaces making pipfile very unreadable and hard to revert all the corruption before committing.

We gave up on trying to preserve the comments/whitespace in the Pipfile a while ago, it was too hard to try and figure out how that ever worked, and it lead to weird conditions where the package name could vary, now it should match the package name being installed.

• And you can no longer manage upgrades by updating pipfile. You always have to remember how to call a command.

The problem is --keep-outdated did a less involved lock resolution process and basically took any new dependency and added it the lock file -- but this was as error prone as anything else, because if it for sure needed an existing dependency to update it would just blindly ignore that fact and you could still end up with uninstallable lock files. It is a command because we cannot detect what has changed in the Pipfile, we need to know what you plan to upgrade and to which category.

• It corrupts pipefile if you don't pass --dev in case of mypy ... You are left with two different versions in lock file (one for non dev, one for dev)

git checkout Pipfile
git checkout Pipfile.lock and retry? That is what I would do ...

• It does not work if you merge from different branches. Because you must manually go over the pipefile and call update manually and still you won't be sure that you didn't miss something.

This was never guarantied to work either in --keep-outdated for similar reasons to above. If you have two PRs that re-lock two different parts of the lock file with some common dependency, you cannot just merge both PRs. Even dependabot would close or rebase its other open PRs when one gets merged that conflicts with a different dependabot PR.

All that being said, I like how you are thinking about the algorithm in the actual proposal. I would be open to accepting PRs to improve this space, but I likely won't have time to work on it this year unfortunately.

[matejsp]

1. That should be fine if you can have clear separation who changes what. Developer changes pipfile, pipenv changes lock file.
2. Yes keep outdated was quite broken. We had CI check to make it less error prone.
3. Yes of course, if even you managed to detect the error. I would prefer pipenv issue a warning in such case.
4. Well it worked quite well with keep outdated because you have only small number of conflicts (if any besides .lock file hash).

I will try to dive in a bit in my free time and see how hard it is to extend it.