Caching mdbook links?

[mark-i-m]

if it is possible to take advantage of mdbook-linkcheck caching now?

Azure has a beta preview of caching, but the rust repo does not use it. Typically s3 buckets are used, but I do not know anything about them or whether it would be appropriate to use.

Originally posted by @ehuss in #66338 (comment)

Splitting this off from the other thread so as not to clutter it.

@pietroalbini Is something like the above possible? Basically, we just want to cache the src/doc/rustc-guide/book/linkcheck directory between builds. This directory is generated by the linkchecker. Doing this would significantly reduce timeouts and other networking issues when linkchecking... We already use this functionality on the rustc-guide repo with Travis caching...

[pietroalbini]

Wouldn't this defeat the purpose of linkcheck?

[mark-i-m]

Hmm that's a valid point. The main reason for putting linked in this CI was to find the PRs that break links in the first place.

If the timeout of cached links is set to ~12 hours, and a build takes 4 hours, then we would reduce the chance of spurious failure by about 2/3, but would know within roughly 3 PRs when the link broke.

[mark-i-m]

cc @rust-lang/wg-learning

[spastorino]

I think caching and maybe having to figure out over a collection of PRs is fine. Shouldn't be that hard.

[Michael-F-Bryan]

I think caching and maybe having to figure out over a collection of PRs is fine. Shouldn't be that hard.

90% of the time you should be able to do a git blame on the offending line and that'll tell you when it was broken.

I think the biggest benefit of caching results is to avoid GitHub's rate limiting. GitHub seems to limit you to, say, a hundred GET requests per minute, whereas the rustc-dev-guide might have several hundred links to GitHub pages. That means you often need to run mdbook-linkcheck to mark+cache the first hundred links as valid and get 429's for the rest, then run it again to mark the next hundred as valid, and so on.

[mark-i-m]

@spastorino @JohnTitor Is there a reason to keep this issue open?

[spastorino]

To be honest I'm not sure what's the current status of things about link checking and caching.

[mark-i-m]

I don't either... I haven't really kept up with it. But given that nothing has happened here for a few years, it seems like this issue is not doing much good by staying open, right?