msrv below Rust 1.16 seemingly cannot be detected

[BurntSushi]

I recently had the occasion to try to find and verify the MSRV of libc. I noticed that cargo msrv told me it was Rust 1.16. What I expected it to be was Rust 1.13. Indeed, building libc with Rust 1.13 works. Building libc with Rust 1.12 fails.

Anyway, here's the output I got:

$ git remote -v
origin  git@github.com:rust-lang/libc (fetch)
origin  git@github.com:rust-lang/libc (push)
$ git rev-parse HEAD
29217b91475c8d7bac444df54defb6163dc138da
$ cargo msrv --version
cargo-msrv 0.15.1
$ cargo msrv
Fetching index
Determining the Minimum Supported Rust Version (MSRV) for toolchain x86_64-unknown-linux-gnu
Using check command cargo check
Check for toolchain '1.30.1-x86_64-unknown-linux-gnu' succeeded

Check for toolchain '1.14.0-x86_64-unknown-linux-gnu' failed with:
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ error: no such subcommand: `check`                                                                                                                                                      │
│                                                                                                                                                                                         │
│       Did you mean `clean`?
     │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Check for toolchain '1.22.1-x86_64-unknown-linux-gnu' succeeded
Check for toolchain '1.18.0-x86_64-unknown-linux-gnu' succeeded
Check for toolchain '1.16.0-x86_64-unknown-linux-gnu' succeeded

Check for toolchain '1.15.1-x86_64-unknown-linux-gnu' failed with:
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ error: no such subcommand: `check`                                                                                                                                                      │
│                                                                                                                                                                                         │
│       Did you mean `clean`?
     │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
   Finished The MSRV is: 1.16.0   ████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ 00:01:14

From looking at the errors, my guess is that this is a result of using cargo check instead of cargo build. Since cargo check was introduced in Rust 1.16, it follows that this method can't be used to reliably detect an MSRV below Rust 1.16.

Ideas:

• Declare this as wontfix because anything before Rust 1.16 is way ancient and not worth supporting.
• When testing Rust <1.16, use cargo build instead of cargo check to see if the build works. It's a bit more wasteful, but is likely to be rare in practice I think?

[foresterre]

Thanks for the report and ideas!

I'll probably opt for the second option :).

[foresterre]

Proposed solutions for checking a wide range of toolchains where some do not support newer options

Problem description

With new Cargo releases come new Cargo subcommands and options. For example, we use cargo check as the default cargo-msrv check command (since cargo-msrv v0.4), which was introduced in Cargo 1.16 (with Rust toolchain 1.16).

These newer options may be desireable to use as cargo-msrv check command, but sometimes you may want to also support building older versions. Currently, this is only possible with backwards compatible cargo commands. For example, if you want to check Rust toolchains prior to Rust toolchain 1.16, you can't use our default check command, but you can run with a custom check command: cargo msrv -- cargo build.

cargo-msrv check command: The command provided to rustup run <toolchain> <command> which determines whether a toolchain will be marked as compatible or not.

Usually we don't really care about the extra steps which cargo build takes over cargo check for the MSRV, and these steps may take quite long (e.g. linking), so when checking a large range which contains older Rust versions which do not support more recent options, we usually do not want to run cargo build on all versions.

That's currently not possible as cargo-msrv requires you to stick to one check command, cargo check by default, or my custom command (e.g. cargo build) as alternative. All toolchains will stick to this one command.

Proposed solutions

Custom check command for a version range

This first solution proposes to let a user specify check commands for a version range; if a RUst version does not fall within the given ranges, we use the default instead.

For example, if we may specify the range <1.16.0, and state that for this range we should override it with the check command cargo build. Then all other version, i.e. >=1.16.0, will fallback to the default cargo check.

We probably can use semver's VersionReq to to parse ranges, and match versions to these ranges.

Fallback check

A variation on the above would be to instead allow setting a fallback command which will be run when the primary check command fails.

For example, if we specify the fallback command to be cargo an-example, and we have a Rust toolchain version which does not support the default cargo-msrv check command, such as Rust 1.15, then upon the failure of running cargo check, we'll also run cargo an-example. If that one does pass, then we'll still consider the cargo-msrv check passed. If both fail, we instead reject.

Fallback check when infering a Cargo "error: no such subcommand"

Same as above, but more restrictive. Instead of always falling back to another command, we only do so when we detect certain Cargo errors.

NB: This would most likely only support certain Cargo commands (normally, anything which can be run via rustup run <toolchain> is accepted.

Hardcoded subsitution

We may also create a reference table of substitutions which take place for certain toolchain versions. E.g. if we detect cargo check with a Rust 1.15 toolchain, we substitute it with cargo build.

This option is probably relatively simple to implement, but it has a considerable disadvantage: flags and options set on the cargo check command may not be 1-to-1 compatible with cargo-build, or similarly to the original issue, they may not be supported yet. We could only do this substitution for the simple base case, or probe cargo check for its supported options, substitute those and then hope for the best 🤞.

Only override cargo-check with cargo-build when not using custom check commands

Another variation would do nothing when a custom check command is specified, but, when the default is used substitute cargo check with cargo build for toolchains <= 1.15. We can add a flag to disable this substitution.

The advantage would be that we would supported toolchains <= 1.15 again, at the marginal extra cost of this substitution.

This would also keep the door open for more complete (and complex) solutions such as the Custom check command for a version range solution above.

Considerations

• Consistency & semantics*: Setting different check commands for different toolchains may offer versatility, but does so at the cost of equal semantics. If a user would specify one range to build the complete workspace while another only builds one package, the semantics aren't shared between runs. It will be up to the user to determine whether this is desirable, and the feature should be used or not.

Remarks

* Semantics may not be the correct word here: I'm trying to state that when this feature would be introduced, the different checks may not have the each an equal effect. For example, cargo build may fail during a linking step, where cargo check would not have.