"Cached" registry rustdoc JSON version mismatch

[thomaseizinger]

Steps to reproduce the bug with the above code

• Cache the target/semver-checks/cache directory in CI
• Wait for a new rustdoc JSON version to be released
• Use new Rust toolchain to build more recent version of rustdoc JSON
• Observe failure: https://github.com/libp2p/rust-libp2p/actions/runs/4378550592/jobs/7663447477#step:14:408

Actual Behaviour

cargo semver-checks fails to read the old JSON.

Expected Behaviour

I can think of multiple approaches:

• Detects that the cached version is stale and re-generates it using the new toolchain
• Incorporate the rustdoc version into the filename (and cache key), effectively invalidating the cache upon upgrades to the Rust toolchain
• Tell people to pin the Rust version they are using in CI
• Avoid the error in the first place: Do the rustdoc JSON versions have to line up? Can't we just use the existing adapters to still run the checks?

Generated System Information

Error appeared in CI so unavailable unfortunately. Version 0.18.3, ubuntu-latest GitHub runner.

Build Configuration

No response

Additional Context

No response

[rinde]

Is there a workaround for this?

[thomaseizinger]

Yes, deleting the cache fixes the problem.

[obi1kenobi]

Apologies for the delay, I was travelling and only just got home. I've started working on a fix in #416.

The recommended workaround in the meantime is clearing the target/cargo-semver-checks/cache directory manually, or running cargo clean (at the expense of a project rebuild).

Do the rustdoc JSON versions have to line up? Can't we just use the existing adapters to still run the checks?

Rustdoc JSON is quite complex and ships bugfixes fairly frequently, so it's better from a correctness and maintenance perspective if the versions match up — it significantly narrows the state space for incoming bug reports on cargo-semver-checks, which not-infrequently produce bug reports upstream. From a purely technical standpoint, though, the versions don't have to match up.

Detects that the cached version is stale and re-generates it using the new toolchain

This is the approach I'm going for. There's no use case I can see where someone would want to semver-check on more than one Rust version — it's always best to check on the latest stable Rust version due to the likelihood of relevant rustdoc JSON bugfixes, so I don't think keeping multiple cached rustdoc JSON versions simultaneously is useful. Let me know if you feel I've missed something here.

[thomaseizinger]

Detects that the cached version is stale and re-generates it using the new toolchain

This is the approach I'm going for. There's no use case I can see where someone would want to semver-check on more than one Rust version — it's always best to check on the latest stable Rust version due to the likelihood of relevant rustdoc JSON bugfixes, so I don't think keeping multiple cached rustdoc JSON versions simultaneously is useful. Let me know if you feel I've missed something here.

The issue with CI caches here is going to be that a modified cache directory won't get saved so we will re-generate the new JSON over and over again until we manually clear it.

We would have to incorporate the version into the cache key for it to automatically invalidate but that is rather tricky to do I think.

[obi1kenobi]

Fixed in #416, will be released shortly.

The issue with CI caches here is going to be that a modified cache directory won't get saved so we will re-generate the new JSON over and over again until we manually clear it.

Would a trick similar to the one used in this repo's CI work?
https://github.com/obi1kenobi/cargo-semver-checks/blob/main/.github/workflows/ci.yml#L163-L172

Essentially, version the cache based on the Rust version. Even if the format version didn't change, it's good to regenerate the rustdocs for new Rust versions in case there are bugfixes in rustdoc that didn't cause a format change — and that cache miss would only happen once every ~6 weeks.

[thomaseizinger]

The issue with CI caches here is going to be that a modified cache directory won't get saved so we will re-generate the new JSON over and over again until we manually clear it.

Would a trick similar to the one used in this repo's CI work? main/.github/workflows/ci.yml#L163-L172

Essentially, version the cache based on the Rust version. Even if the format version didn't change, it's good to regenerate the rustdocs for new Rust versions in case there are bugfixes in rustdoc that didn't cause a format change — and that cache miss would only happen once every ~6 weeks.

Yep that should work!

[thomaseizinger]

I ended up incorporating the tool-version too: libp2p/rust-libp2p#3664

[obi1kenobi]

Released in v0.19.0: https://github.com/obi1kenobi/cargo-semver-checks/releases/tag/v0.19.0