Support pdm outdated

[ghost]

Is your feature request related to a problem? Please describe.

There is no any related problem.

Describe the solution you'd like

1. command pdm outdated, or pdm list --outdated.
2. pdm prints all the dependencies (in local/global) which is outdated.

Reason:
It's common feature of package manager and I used them often.
If this proposal is positive, I will try implementing and send PR.

Remarks:
I think pdm outdated is better than pdm list --outdated because I want an exact one option.

• pdm outdated -> returns only top-level dependencies.
• pdm outdated -a(--all) -> returns all dependencies.

Motivation came from this issue python-poetry/poetry#2684 (comment).

[frostming]

Some points need more discussion:

1. Reuse existing commands instead of introducing a new one. I would suggest pdm update --dry-run where you can use sections or packages arguments to limit the update range. The --dry-run flag means to relock and show the difference without modifying the files on the disk.
2. There are three version sets: one is the latest version on the PyPI or the new version after relock, another is the locked version, and one is the installed version. By using this command, which two are we going to compare between?

[frostming]

The design will be as the following:

• add --dry-run to update command to display (to install, to update, to remove) packages but not to modify the pdm.lock.
• Under the hood, lock will be performed to fetch the latest available versions, then show the diff between the result and the installed versions in the current environment
• Other arguments and options such as --dev, sections, packages, --unconstrained and --update-strategy will all influence the result.
• Based on the above, add an option --top to limit the result to top-level packages only(those list in pyproject.toml), again, which can be further limited by --dev, sections. Plus, when --top is passed, specifying packages via packages is not allowed.

If anything is not clear, feel free to reply in this thread

[ghost]

It's all clear, thank you for giving me advice (sorry for replying late, I'm at working currently).

I'll try one when I have a time.

[frostming]

Never mind, I can implement it if you are not able to.

[frostming]

Done, now you can test on the master branch. Note that --outdated is just an alias of --dry-run

[ghost]

Wow, thank you very much. It's great.

[danielmoessner]

I would prefer to have a direct pdm outdated command. This command should not only list the packages pdm WOULD update but also the packages that in general CAN be updated.

In my opinion tools like pdm should help the developer as much as they can. I guess this is also one of the reasons pdm became successful because for me it is easier to work with than pipenv or pip + venv.

Creating a shell script or coming back to this thread is just annoying as the command is too long to be easily remembered.

On the technical side it seems really easy to make pdm call its other --dry-run --unconstrained --top command on pdm outdated.

[AvnerCohen]

Agree on that.
The "outdated" command should simply be a shortcut to showing the currently locked version of each lib vs the latest one on pypi.

[frostming]

The "outdated" command should simply be a shortcut to showing the currently locked version of each lib vs the latest one on pypi.

The problem is that the latest version on PyPI may not satisfy the current dependency specifications, so you will never be able to upgrade to that version unless you modify pyproject.toml. I am afraid this will bring some anxiety and confusion.

[danielmoessner]

Good point how about 4 columns?

>> pdm outdated

package | current_version | would_update_to   | latest_version
--------------------------------------------------------------
django  | 4.0.1           | 4.0.10 (==4.0.*)  | 4.2.8
fastapi | 0.96.1          | 0.99.0 (<0.100.0) | 0.104.1

[AvnerCohen]

I suggest to work based on existing solutions:
https://classic.yarnpkg.com/lang/en/docs/cli/outdated/

[frostming]

There are 5 levels of versions available for a specific package:

1. The version currently installed in the working set
2. The version pinned in the lockfile
3. The version to update if we run pdm update WITHOUT modifying the pyproject.toml
4. The version to update if we run pdm update --unconstrained(ignore the specifiers).
5. The latest version on PyPI.

Note 4 and 5 can be different, because transitive dependencies can constrain each other even we remove all version specifiers.

I'm not sure what versions users care the most and if users can understand the meaning of these versions. And the yarn/pnpm solution seems too simple to me.

It's worth to mention that pdm update --dry-run compares 1 and 3/4, depending on whether --unconstrained is passed.

[sanmai-NL]

@frostming Is it correct that PDM currently does not support updating the pyproject.toml version specifiers to the minimum saved in the lockfile (--save-minimum)?

[AvnerCohen]

@frostming I would go MVP here if at all possible.
So of the 5 levels, you mentioned:

1. The version currently installed in the working set
2. The version to update if we run pdm update WITHOUT modifying the pyproject.toml
3. The latest version on PyPI.

I suspect this will be the cheapest to calculate and should help determine if there is further need for additional effort on this.
I would even suggest if it's easier to skip 3 at phase 1 if at all helpful.

[frostming]

I would even suggest if it's easier to skip 3 at phase 1 if at all helpful.

So you suggest 1,3,5 as listed in my previous comment. My point is pdm update --dry-run already covers 1, 3 so a dedicated command is not of much benefit in this case.

[AvnerCohen]

I respectfully disagree - it's the 5 that is the main part missing for me.
On a long running project, the cycle of updating packages and making sure you are not getting too much behind isa never ending cycle.
If I want to answer the question: "How far am I from the latest version of this package", there is no way to do that right now, outside of going into pypi for each single package.

[danielmoessner]

For me personally it is also 5 which is the most useful and interesting.

[frostming]

This function is not difficult to implement at all. It's just a matter of requesting new versions from PyPI. However, I tend to avoid introducing a command that overlaps too much with other command functions. In addition to the update --outdated command, pdm list does something similar, to list the installed versions, we can just add another column to the output, instead of adding a new command. Right?

[AvnerCohen]

I know very little about the overhead of each approach, so would be hard for me to take a clear stance here, here is what I think:

1. update --outdated --dry-run or any such direction is not amazing because it overloads the "update" (POST/PUT) command. The command, we are looking at an informational (GET) zero change command here, so I think it's important to keep it like that.
2. pdm list is clearly informational so fits the bill, but will it be added with an extra flag (--outdated) ? because obviously we do not want the base all-local command pdm list to require also network call to pypi by default. This makes it a tad less trivial.
3. Looking at other package managers, we see the "outdated" used as the informational command for this need:
   NPM (NodeJS) - https://docs.npmjs.com/cli/v10/commands/npm-outdated
   YARN (NodeJS) - https://classic.yarnpkg.com/lang/en/docs/cli/outdated/
   Cargo (Rust) - https://github.com/kbknapp/cargo-outdated
   Composer (PHP)- https://getcomposer.org/doc/03-cli.md#outdated
   Bundler (Ruby) - https://fig.io/manual/bundle/outdated

On the other hand:
Golang overloads the "list" command - https://github.com/golang/go/wiki/Modules#how-to-upgrade-and-downgrade-dependencies

So, with that in mind, I guess the trade off is between, as you said "avoid introducing a command that overlaps too much with other command functions" , Familiarity of the interface and Actual effort for the change.

My vote would be for a clear and simple pdm outdated followed by a pdm list --outdated.

[frostming]

It seems most people in this thread prefer pdm outdated, now let's go this path. Adding a new command needs more effort than just implementing the required function. We need to consider what is the default behavior and what options we offer to tweak it. The following is some points of my thoughts:

The output should contain 4 columns, listing the package name, installed version, the latest possible version that satisfy the constraints, and the latest version on PyPI, respectively.

Package Installed Constrained Latest
------- --------- ----------- ------

Here comes the first question:

What packages should be listed in the table?

We will likely include the -G/--no-default/--prod option group to filter different groups in the lockfile. But what should we do about those packages not listed in the lockfile? Should they be added to the list of packages, with the Constrained value as <none>, or should they be skipped? If the former, how should the -G/--no-default/--prod option affect this?

Secondly, we can support -u/--unconstrained to change the versions displayed in the 3rd column, similar behavior to update and add commands.

Thirdly, we can support passing a package name as the argument: pdm outdated <package> to limit the content to the selected package only, ideally a glob pattern can be also accepted.

Fourthly, we'll support --format option like pdm list, possible values are table and json currently.

Anything to add?

[AvnerCohen]

That's amazing!
I don't think I have enough knowledge on the internals and I wouldn't want to waste your time side issues.
I am guessing the defaults should be same as for list or update, if this makes sense.

But what should we do about those packages not listed in the lockfile?
Can't the same logic as in "list" apply here as well?

Not sure if helpful, but the yarn output is pretty elaborated and provides extra data ("Package Type") to add more context on top fo the package:

[sanmai-NL]

I only see value in the URL column on top of what @frostming proposed. The URL column can be used to quickly check out what wheels/artifacts have been released.

This also begs the question, what does latest mean? I often see that there are no compatible binary wheels for some package, e.g. none for RCs as opposed to stable versions.

[frostming]

Can't the same logic as in "list" apply here as well?

No, pdm list lists either installed packages, or pinned packages when --resolve is passed, but don't list both, which is the case of pdm outdated, so the installed but not pinned ones belong to a grey zone.

[AvnerCohen]

Thanks for the clarification.
So, If I get this right, "installed but not pinned" means libs listed with "*" in their version?

[frostming]

So, If I get this right, "installed but not pinned" means libs listed with "*" in their version?

Nope, "pinned" means recorded in the lockfile. There might be packages that are not in the lockfile but installed by, for example, pip.

[AvnerCohen]

I see. In such case I suspect these are mainly edge-cases residuals (pip install behind the back of PDM, an incomplete deletion of a remove dependency, an upgrade that no longer depends on a sub-dependency and such),
In my mind, and for the specific usage of "outdated", I would ignore it all together if possible.
In a way, PDM should report "outdated" on the packages he is "owning" if somethign happened out-of-band, sounds less relevant to me.