Make cargo outdated part of cargo itself

[behnam]

In Rust ecosystem, one main concern for library developers is to not have semver breaking updates to their package, because of the fear that majority of the dependent packages not getting the routing updates that would follow a breaking update.

There are two general solutions to this problem:

1. The old-school back-porting of updates, so that dependents on any API version would get all the routine updates. This solution can easily become a huge maintenance cost on library developers and is not a common practice in the Rust community at the moment.

2. Help owner of dependent packages to update their dependency versions and apply the changes needed, if any. This solution can be much less time consuming, and distributes the workload throughout the dependency chain. It also means that only the versions being used get the maintenance work needed, but not all theoretically-possible versions. So, in rustacean words: you pay for what you use.

Based on this, I want to propose to add new features to help owners with keeping their dependencies up-to-date. Most of the functionality can be exposed via the cargo update command.

Proposed Features

• Warn on dependencies stuck in older versions. This would be a plain warning on cargo update. The feature would be enabled by default, but there will be an option to disable it; and there can be one release version of Cargo as transition period, where the option is disabled by-default, but there's a notice about the upcoming change, if it applies.

• Commands to auto-update some/all dependency versions to latest. This option would update the relevant manifests to use a latest version of a single or multiple packages. This can also be an option to cargo update, similar to --precise, but instead of updating the lock file, it would also update the manifest with the latest version, or some user-provided version.

  This is a tool to update Cargo Manifest files. Currently, cargo update only works with the Cargo Lock file, so it may be a good idea to have a separate sub-command for the manifest manipulation features. But since it's a core functionality, it's better have this functionality in Cargo proper.

• Auto-update versions to a successful build. This is something that we can aim for in longer term, to have cargo update dependencies, and in case of failures, bisect and find the most-recent version that does not break the builds and/or tests. Best place for this may be a third-party cargo command and not cargo itself.

What do you think?

[behnam]

I'm sure there are many examples where these can be helpful. Here's one that inspired me to write up this proposal: servo/unicode-bidi#46 (comment)

/cc @alexcrichton, @Manishearth, @anowell

[alexcrichton]

I think we'd probably want to canvas what other package managers do here, e.g. do npm, yarn, bundler, rubygems, pip, etc do anything like this? I was personally under the impression that this sort of concerns was typically farmed out to a third-party service.

[behnam]

Here’s a summary of similar features in some other package managers.

---

JS NPM

NPM CLI has built-in outdated subcommand that shows the status of packages that their current, wanted and latest versions do not match.

Like cargo update, the npm update does update the current version to wanted, if needed and possible.

It looks like that npm update does not have any more functionality in this area. Also, no other subcommand look related.

Ref: https://docs.npmjs.com/cli/outdated

Description

(from https://github.com/npm/npm/blob/latest/doc/cli/npm-outdated.md)

This command will check the registry to see if any (or, specific) installed packages are currently outdated.

In the output:

• wanted is the maximum version of the package that satisfies the semver range specified in package.json. If there's no available semver range (i.e. you're running npm outdated --global, or the package isn't included in package.json), then wanted shows the currently-installed version.

• latest is the version of the package tagged as latest in the registry. Running npm publish with no special configuration will publish the package with a dist-tag of latest. This may or may not be the maximum version of the package, or the most-recently published version of the package, depending on how the package's developer manages the latest dist-tag(1).

• location is where in the dependency tree the package is located. Note that npm outdated defaults to a depth of 0, so unless you override that, you'll always be seeing only top-level dependencies that are outdated.

• package type (when using --long / -l) tells you whether this package is a dependency or a devDependency. Packages not included in package.json are always marked dependencies.

Example

Edit package.json and add "underscore": "1.7" to dependencies list.

$ npm outdated
Package     Current  Wanted  Latest  Location
underscore  MISSING   1.7.0   1.8.3  underscore

$ npm install
underscore@1.7.0 node_modules/underscore

$ npm outdated
Package     Current  Wanted  Latest  Location
underscore    1.7.0   1.7.0   1.8.3  underscore

---

JS Yarn

Yarn CLI has built-in outdated subcommand to “checks for outdated package dependencies”.

Ref: https://yarnpkg.com/lang/en/docs/cli/outdated/

Description

Lists version information for all package dependencies. This information includes the currently installed version, the desired version based on semver, and the latest available version.

For example, say your package.json has the following dependencies listed:

"dependencies": {
  "lodash": "4.15.0",
  "underscore": "~1.6.0"
}

The command run should look something like this:

yarn outdated
Package    Current Wanted Latest
lodash     4.15.0  4.15.0 4.16.4
underscore 1.6.0   1.6.0  1.8.3 
✨  Done in 0.72s.

---

Ruby Gems

Gems CLI has built-in outdated subcommand to “display all gems that need updates”.

Ref: http://guides.rubygems.org/command-reference/#gem-outdated

Description

(From ref.)

The outdated command lists gems you may wish to upgrade to a newer version.

You can check for dependency mismatches using the dependency command and update the gems with the update or install commands.

---

Ruby Bundler

Bundler CLI has built-in outdated subcommand to “list installed gems with newer versions available”.

Ref: https://bundler.io/v1.11/bundle_outdated.html

Description

(From ref.)

Outdated lists the names and versions of gems that have a newer version available in the given source. Calling outdated with [GEM [GEM]] will only check for newer versions of the given gems. By default, available prerelease gems will be ignored.

---

Python PIP

PIP CLI's list subcommand has an --outdated option, to "list outdated packages", and an --uptodate option to "list uptodate packages".

Ref: https://pip.pypa.io/en/stable/reference/pip_list/

Examples

List installed packages.

$ pip list
docutils (0.10)
Jinja2 (2.7.2)
MarkupSafe (0.18)
Pygments (1.6)
Sphinx (1.2.1)

List outdated packages (excluding editables), and the latest version available.

$ pip list --outdated
docutils (Current: 0.10 Latest: 0.11)
Sphinx (Current: 1.2.1 Latest: 1.2.2)

[behnam]

In summary:

1. JS and Ruby package managers have built-in outdated subcommand to list packages "not in good standing", in the meaning that their installed version is not what's asked for, or what's asked for is not the latest upstream version (not necessarily the largest one, but the latest marked as stable release).

2. Python PIP has similar functionality via the built-in list subcommand, which makes it more generic, but harder to discover and use for beginner/average users.

3. Looks like all these commands are readonly and have no built-in functionality to update manifest files to the latest version.

4. I have not seen signs of any warning-like functionality regarding out-of-date packages in other commands.

[behnam]

So, IMHO, we can plan to for these functionalities, applicable only to direct dependencies of a package/workspace.

a) A built-in outdated command to help developers find outdated direct dependencies. This would be the readonly and the human-readable output will be similar to the JS/Ruby package managers.

b) Expand (a) to support workspaces.

c) Add options to outdated to make it read-write, updating version numbers of one/more/all outdated dependencies. This, I assume, would be the only/first cargo command to write to the manifest, so would need more design and implementation work.

d) Expand (c) to also support workspaces.

These two can be enough for shell scripts and third-party cargo commands to easily build automations for stable upgrades, meaning that run the tests with single upgrades and revert if there's compile or test errors.

What do you think?

[steveklabnik]

https://crates.io/crates/cargo-outdated works well today

[behnam]

Unfortunately, cargo-outdated package doesn't have any support for workspaces, yet.

Also, a question would be about the original proposal to add optional warning (possibly as an opt-in, via env-var and/or cli option) for cargo update about existence of outdated package(s). I'd like to know how everyone feels about that.

[alexcrichton]

I'd be totally down for adding native outdated support in Cargo itself, but we'd probably want to work closely with the upstream author of cargo-outdated to ensure we coordinate. I know I use it often enough!

[alexcrichton]

Also thanks so much for doing that investigation @behnam, it's quite helpful!

[Keats]

A big +1 for getting cargo outdated built-in.

I've been using cargo-outdated for ages now but as @behnam says, it doesn't support workspaces yet and I use so much with yarn and pip that I believe it should be part of Cargo.

Just to update the yarn output, it actually looks like the following now:

Package            Current       Wanted  Latest  Package Type    URL                                                       
@types/lodash      4.14.70       4.14.71 4.14.71 devDependencies https://www.github.com/DefinitelyTyped/DefinitelyTyped.git
@types/react       15.0.38       15.0.39 15.0.39 devDependencies https://www.github.com/DefinitelyTyped/DefinitelyTyped.git
gulp               4.0.0-alpha.2 exotic  exotic  devDependencies gulpjs/gulp#4.0                                           
signature_pad      2.2.0         2.2.1   2.2.1   dependencies    https://github.com/szimek/signature_pad                   
webpack-dev-server 2.5.1         2.6.1   2.6.1   devDependencies http://github.com/webpack/webpack-dev-server              
Done in 2.27s.

Having the URL in the output is quite nice to be able to see the changes too. Separating the dev-dependencies, build-dependencies and dependencies more cleanly than yarn does would be nice too but then it might become too complex to display with workspaces.

One change to do compared to cargo-outdated that is already mentioned is that imo, it should only list direct dependencies of the project, not subdependencies. Here's an example on one of my projects:

    Name             Project Ver  SemVer Compat  Latest Ver
    hyper               0.10.12        --          0.11.1
    hyper->base64       0.5.2          --          0.6.0
    hyper->mime         0.2.6          --          0.3.2
    hyper->unicase      1.4.2          --          2.0.0
    slug->unidecode     0.2.0          --          0.3.0

I only care about hyper in that list.

If we want to go c) I would recommend checking the yarn upgrade-interactive command as an example of how to handle it.

I'd be interested to do the implementation if that ends up being accepted for cargo!

[behnam]

Good ideas, @Keats!

Having the URL in the output is quite nice to be able to see the changes too.

Agreed.

Also, on a similar note, a --verbose option that shows you also all in good standing, telling you what's been in the check list.

Separating the dev-dependencies, build-dependencies and dependencies more cleanly than yarn does would be nice too

I'm not sure what build-dependencies is here. But, about dev-dependencies, agreed that sometimes it's not a concern and better be able to leave it out of calculation. So, maybe we need a set of options to disable each group of these dependencies.

but then it might become too complex to display with workspaces.

Actually, I think the readonly functionality won't be that complex for workspaces, since there's expected to be one Cargo.lock already.

But, it's a good reminder that we also need something like NPM's location, which shows which local (in the workspace) package is depending on the old version.

And as a plus, this would be Cargo's internal tool to warn on a workspace (or even a package) having direct dependencies on multiple versions of a single package.

it should only list direct dependencies of the project, not subdependencies.

Exactly. I'm not sure if we want a feature to perform these checks more than one level depth. I think we can start with direct deps and see if there's demand for an option to extend.

I'd be interested to do the implementation if that ends up being accepted for cargo!

Awesome! I won't be able to get to this in the next few weeks, but can help with some tests and reviews, if needed.

[behnam]

So, looks like we have general consensus on the core functionality of a built-in outdated subcommand and the rest can be discussed per PR, I assume.

Also, let's CC author of cargo-outdated. @kbknapp, what do you think about these?

[Keats]

I'm not sure what build-dependencies is here. But, about dev-dependencies, agreed that sometimes it's not a concern and better be able to leave it out of calculation. So, maybe we need a set of options to disable each group of these dependencies.

Build dependencies would be bindgen for example: https://github.com/compass-rs/sass-rs/blob/master/sass-sys/Cargo.toml#L16
Dev dependencies is important to display in JS-land since I have more dev dependencies than actual dependencies but I think it still makes sense to display outdated info for all crates listed in my Cargo.toml by default. We could have a flag to disable build/dev dependencies but I don't think it would be used very often: if you run cargo outdated, you usually want to see all of the crates.

Actually, I think the readonly functionality won't be that complex for workspaces, since there's expected to be one Cargo.lock already.

My worry was more on how to display it well in the terminal when you have many workspaces like habitat. I guess scrolling a bit is not a huge issue

[behnam]

Agreed, @Keats. :)

Funny thing about build-dependencies: I could tell what it is, but didn't find it anywhere under the manifest reference: http://doc.crates.io/manifest.html. Turns out, it's only mentioned in this other page: http://doc.crates.io/specifying-dependencies.html#build-dependencies I'll submit a PR to add it to the list in http://doc.crates.io/manifest.html#dependency-sections so it doesn't happen again.

[behnam]

@alexcrichton, is this something that needs an RFC, or we can go ahead with the general consensus here and start the implementation?

If no RFC, I suppose we can create a new tracker issue for the built-in cargo outdated subcommand with the details, and leave this issue for discussion of other related matters, like the proposed warning on cargo update when there are packages staying behind.

[kbknapp]

Sorry all, I just saw this! I'm totally fine with adding a native cargo outdated command! 😄 I haven't had much time to work on my open source projects lately due to my day job, but @Frederick888 has stepped up to help with cargo-outdated specifically.

The biggest thing I see that needs to happen for cargo-outdated (native or third party) is to add workspace support.

@Keats

I only care about hyper in that list.

cargo-outdated (third party subcommand) currently has the -R, --root-deps-only or --depth=# options that does exactly what you're looking for.

There's also -r, --root=PKG to treat a particular package as the root, and finally -p, --package=PKG to only look for updates for a particular package/crate. All of which should be combinable as well.

---

Slightly off topic, this sort of brings up the topic of third party subcommands moving closer to native commands, and if there's an official method or place for this to happen? Similar to how libs move to the nursery, is there a place where third party subcommands can move closer to cargo itself, perhaps even distributed by default? There are times where that may be less work than re-implementing functionality. I understand with outdated it may be easier to just duplicate the functionality inside cargo, but I highly doubt this is the last time this sort of situation will come up. Just thinking out loud. 😉

[lnicola]

I only care about hyper in that list.

cargo-outdated (third party subcommand) currently has the -R, --root-deps-only or --depth=# options that does exactly what you're looking for.

There are two reasons you might have an outdated dependency of a crate you use. Either the author hasn't upgraded yet, or you have the old version in your Cargo.lock file. cargo-outdated's SemVer Compat column should probably help here, but I could never figure out how it works, e.g.:

    Name                  Project Ver  SemVer Compat  Latest Ver
    rocket->base64           0.6.0        0.5.2           --

So from time to time I remove my Cargo.lock to get a new one.

[Frederick888]

@lnicola

I could never figure out how it works

We have just made a new release. Could you please test it again?

@kbknapp

Slightly off topic, this sort of brings up the topic of third party subcommands moving closer to native commands, and if there's an official method or place for this to happen?

Good point. Actually when I was refactoring cargo-outdated I struggled a lot about whether to add cargo as a dependency as it tended to be kind of an overkill (and would cost me plenty of time to read the documents). But now if we are talking about embedding the functionality into cargo itself, which is something that I'm also happy with, apparently depending on cargo is a better choice.

So yeah, it would be better to have such a guideline, or even a list of subcommand development suggestions to developers.

[lnicola]

I was literally looking for a new version when you wrote that 😀.

Before:

    Name                  Project Ver  SemVer Compat  Latest Ver
    rocket->base64           0.6.0        0.5.2           --
    r2d2-diesel->diesel      0.15.1       0.15.2        0.15.2
    rocket->hyper            0.10.12        --          0.11.2
    time->libc               0.2.28       0.2.29        0.2.29
    uuid->rand               0.3.15       0.3.16        0.3.16
    time->redox_syscall      0.1.27       0.1.29        0.1.29
    toml->serde              1.0.10       1.0.11        1.0.11
    rocket->smallvec         0.4.1          --          0.2.1
    rocket_contrib->tera     0.10.8       0.10.9        0.10.9
    rocket->toml             0.4.2        0.4.4         0.4.4

After:

Name                         Project Ver  SemVer Compat  Latest Ver
backtrace->libc                 0.2.28       0.2.29        0.2.29
backtrace-sys->libc             0.2.28       0.2.29        0.2.29
diesel                          0.15.1       0.15.2        0.15.2
diesel_codegen->diesel          0.15.1       0.15.2        0.15.2
diesel_infer_schema->diesel     0.15.1       0.15.2        0.15.2
isatty->libc                    0.2.28       0.2.29        0.2.29
memchr->libc                    0.2.28       0.2.29        0.2.29
num_cpus->libc                  0.2.28       0.2.29        0.2.29
r2d2-diesel->diesel             0.15.1       0.15.2        0.15.2
rand->libc                      0.2.28       0.2.29        0.2.29
rayon-core->libc                0.2.28       0.2.29        0.2.29
rayon-core->rand                0.3.15       0.3.16        0.3.16
ring->libc                      0.2.28       0.2.29        0.2.29
rocket->toml                    0.4.2        0.4.4         0.4.4
rocket_contrib->serde           1.0.10       1.0.11        1.0.11
rocket_contrib->tera            0.10.8       0.10.9        0.10.9
serde                           1.0.10       1.0.11        1.0.11
serde_json->serde               1.0.10       1.0.11        1.0.11
tera->serde                     1.0.10       1.0.11        1.0.11
time->libc                      0.2.28       0.2.29        0.2.29
time->redox_syscall             0.1.27       0.1.29        0.1.29
toml->serde                     1.0.10       1.0.11        1.0.11
uuid->rand                      0.3.15       0.3.16        0.3.16

After removing Cargo.lock:

All dependencies are up to date, yay!

👍

[behnam]

After removing Cargo.lock:

@lnicola, are you saying that cargo update doesn't give you the same results? If so, can you be a bit more specific on in which circumstances that happens?

[lnicola]

@behnam I actually had a copy of it, so I just tried cargo update on it. Looks like the results were the same ("All dependencies are up to date").

If you're asking why I said I remove Cargo.lock instead of running cargo update, there's no particular reason.

[lnicola]

@Frederick888 Looking at my output, I think semver-breaking updates for dependencies are not reported anymore?

    rocket->hyper            0.10.12        --          0.11.2

[Frederick888]

@lnicola Actually it wasn't supposed to be reported. Previously if a package has multiple occurrences with different versions/semver requirements in the dependency tree, the program may misbehaviour from time to time.

cargo-outdated checks the "latest" version by replacing the semver requirements of direct dependencies with "*", which means it doesn't really give you the "latest released" versions of indirect ones. This is sometimes not quite intuitive and there's a related issue for this (kbknapp/cargo-outdated#25).

[lnicola]

Previously if a package has multiple occurrences with different versions/semver requirements in the dependency tree, the program may misbehaviour from time to time.

I don't think that was the case for hyper in my example.

checks the "latest" version by replacing the semver requirements of direct dependencies with "*", which means it doesn't really give you the "latest released" versions of indirect ones.

I see. It might be useful to have those reported, so you know that your dependencies use older crates. But I don't feel too strongly about it.

[behnam]

Regarding making it easier for third-party libraries to use cargo internals, and make it easier to bring in third-party commands as built-in: maybe splitting cargo into smaller crates (in one repo, set up as workspace) would help?

Basically, I think reading and validating the manifest (with understandings of workspace and various dependencies) should be a separate library from the commands consuming the manifest to take actions.

[jonhoo]

I believe some of the confusion about "why doesn't cargo-outdated show me X" also stems from kbknapp/cargo-outdated#134, which I just stumbled upon.

[svenstaro]

Is there any current progress on this? Does anyone know what the last status of this was in upstream?

[ffimnsr]

Do we have an update for this? I like cargo-outdated though I think there should be a standardized sub-command inside cargo that checks package that are outdated. As new users would need to search for package just to run that certain functionality

[jonhoo]

Just leaving a note here that cargo tree was recently adopted into cargo proper in #8062, which, at least to me, indicates that it's not impossible the same may happen to cargo outdated if someone does the work.

[kbknapp]

cc @deg4uss3r

[deg4uss3r]

Thanks for the cc @kbknapp I'm in the middle of wrapping up at my current job, but I should be more free starting next week to really look into this :)

[svenstaro]

Hey, have you been able to look into this perchance?

[deg4uss3r]

@svenstaro Yes I have! Progress is moving slowly, but I am working on it :) starting the new job at the same time is tough though

[svenstaro]

Ok sweet, thanks for the update!

[svenstaro]

@deg4uss3r friendly ping :D

[deg4uss3r]

@svenstaro Thank you very much for keeping me honest!

So, I am working on this; however, I have been doing so offline because I see a lot of little things to improve on cargo-oudated that I would like to simplify as a cargo sub-command, but still possibly maintain cargo-outdated as a heavier project based option. So here's my work out in the open: https://github.com/deg4uss3r/outdated2

This will look VERY rough right now, I'm sorry it's been a pretty interesting 2020 so far!

So this simplifies the project way down to only looking at workspace dependencies and requesting the latest from https://crates.io through the API. I think this is the best choice to add to cargo since we want it to always work no more compile issues with conflicting dependencies and it will only check the outdated dependencies of the current project rather than diving into dependencies that the launcher will have no control over.

There's A LOT (as you can tell) that I need to work on the biggest is integrating with cargo directly, but I have been paying close attention to the dependencies that cargo uses to make sure I'm reusing what is already imported. So far I see that I need to really flesh out before actually putting into cargo for the MR:

• Only report updates on the channel the dependency is in (e.g. don't offer a beta when the user is using a stable version)
• Proper error handling
• Command line integration for options
• support for outputting json

I would love some opinions on this scaled down version before I continue further though as, personally I think this is what we want as a cargo sub-command, but I might be totally in the minority there.

Edit: I'm sure there's some really simple optimizations I haven't done yet but right now it's pretty fast, it takes about 9 seconds (release build, from rustc stable 1.46) for servo... which isn't bad for something like this!

[svenstaro]

Great to see some progress!

Since you asked for feedback: Features that I really like in the current version: --ignore, --depth, --workspace. --exit-code is interesting for CI.

[golddranks]

I use cargo outdated -R/--root-deps-only quite often, because the standard output is a bit verbose.

[dessalines]

I suggest looking at yarn upgrade-interactive --latest for a really clean upgrading interface. I often use cargo outdated, but having to change every dep manually is a pain.

[Zymlex]

To begin cargo outdated must support all cargo modes.

[epage]

btw #10498 exists for moving cargo-edit's cargo-upgrade into cargo.

I wonder if cargo outdated could be a resolved via cargo update and cargo upgrade via

• Dry-run support
• An alternate output format

To begin cargo outdated must support all cargo modes.

Could you clarify which "modes" you are talking about that it'd need to support

[epage]

cargo upgrade v0.10.3 now provides a table summary of upgrades. This doesn't completely replace cargo outdated as this only handles root dependencies. I still hope we could have cargo update handle that case.

Example for clap:

$ cargo upgrade --dry-run
    Updating 'https://github.com/rust-lang/crates.io-index' index
    Checking clap's dependencies
name              old req        locked latest new req        note
====              =======        ====== ====== =======        ====
clap_derive       =4.0.0-alpha.0 -      -      =4.0.0-alpha.0 pinned
clap_lex          0.2.2          -      -      0.2.2
bitflags          1.2            1.3.2  1.3.2  1.2            compatible
textwrap          0.15.0         0.15.0 0.15.0 0.15.0
unicase           2.6            2.6.0  2.6.0  2.6
indexmap          1.0            1.9.1  1.9.1  1.0            compatible
strsim            0.10           0.10.0 0.10.0 0.10
atty              0.2            0.2.14 0.2.14 0.2
termcolor         1.1.1          1.1.3  1.1.3  1.1.1          compatible
terminal_size     0.1.12         0.1.17 0.2.1  0.2.1
backtrace         0.3            0.3.66 0.3.66 0.3
once_cell         1.12.0         1.13.0 1.13.0 1.12.0         compatible
trybuild          1.0.18         1.0.63 1.0.63 1.0.18         compatible
rustversion       1              1.0.8  1.0.8  1
trycmd            0.13           0.13.4 0.13.4 0.13
humantime         2              2.1.0  2.1.0  2
snapbox           0.2.9          0.2.10 0.2.10 0.2.9          compatible
shlex             1.1.0          1.1.0  1.1.0  1.1.0
static_assertions 1.1.0          1.1.0  1.1.0  1.1.0
note: Re-run with `--pinned` to upgrade pinned version requirements
note: Re-run with `--to-lockfile` to upgrade compatible version requirements
warning: aborting upgrade due to dry run

(actual output is colored to make it easier to scan)

[tgross35]

If this gets picked up, one reasonable change from cargo-outdated is that having outdated packages should exist nonzero by default. It is a bit surprising when running in CI that you need to use --exit-code 1 to make it fail

[pitaj]

Just wanted to note here that cargo upgrades also exists and seems to be more compatible (works on rust-lang/rust workspace for instance).

[epage]

While its not as fancy as cargo upgrade or cargo outdated, #13372 added highlighting of non-latest dependencies to cargo.