Dist index gone

[jethrogb]

These indices are out of date:

• https://static.rust-lang.org/dist/index.html
• https://static.rust-lang.org/dist/index.txt

Edit: they're 404's now.

[jethrogb]

There was some discussion on this in the infra channel link link

Main takeaways from that discussion:

• generating the index was taking to long, the recommendation is to use the AWS S3 List Objects API
• listing all objects using the AWS S3 List Objects API takes a very long time because you'll be rate-limited after a few requests
• rustup currently doesn't use the index, but also rustup currently doesn't have a way to list which toolchains are installable

[jethrogb]

I think there should be an index of at least all available toolchains, and preferably also for each toolchain which components exist.

[hellow554]

The links above are dead

[jethrogb]

The index.txt is now, index.html is still working for me, but may be cached?

Edit: yup, getting dead links now too.

[acmcarther]

I'm not sure if this is the main thread of this issue, but hopefully an observer here can redirect me if not.

Is this comment saying that removing the index was deliberate? That index is the suggested means of finding older releases on the other installation methods page.

Past releases can be found in the archives.

[jethrogb]

@estebank T-doc? Should be T-infra

[Mark-Simulacrum]

This is expected behavior, and movement here is unlikely at this point so I'm going to close.

[jethrogb]

@Mark-Simulacrum why is movement unlikely? IIRC @pietroalbini had some thoughts on this.

[Mark-Simulacrum]

There are options to explore, but they're all pretty internal and this issue won't really help with tracking them I think. I suppose we could move it to the infra team repo.

[SimonSapin]

A work-around is to run AWS Command Line Interface. However it takes a while and as far as I can tell there is no server-side glob filtering. When possible, requesting a dated sub-directory is more reasonable.

$ time aws --no-sign-request s3 ls static-rust-lang-org/dist/ | wc -l
61253

real	0m45.545s
user	0m21.830s
sys	0m1.977s

$ time aws --no-sign-request s3 ls static-rust-lang-org/dist/2019-09-02/ | wc -l
1998

real	0m4.342s
user	0m1.495s
sys	0m0.143s

[jethrogb]

@SimonSapin I had written this limitation:

• listing all objects using the AWS S3 List Objects API takes a very long time because you'll be rate-limited after a few requests

I don't remember exactly what I was trying to do here but it's not clear to me why you're not running into the same issue.

[SimonSapin]

45 seconds for this feels like a very long time to me. When I don’t pipe the output it seems to come in bursts. Maybe the cli tool rate-limits itself.

[jethrogb]

45 seconds for this feels like a very long time to me.

True, but the way I was remembering it, it was in the minutes or tens of minutes.

[Mark-Simulacrum]

So, to be clear, the list command there is issuing hundreds or thousands of API queries so it'll likely be slow -- the solution here is to use the S3 Inventory API but no one has gotten around to implementing that just yet.

Part of the reason why is that no super solid reasons for wanting to list objects in the whole bucket have been provided yet; we do have index.html (which internally uses list API, via JS, IIRC) in each subdirectory of the static.rust-lang.org bucket.

[jethrogb]

The full list is needed to know all nightlies/releases that exist, e.g. for bisecting or archiving.

[Mark-Simulacrum]

Both bisection tools that I've written simply run forwards by single-day increments after binary search hits a spot if they can't download nightlies which works well.

Is there any need for all the files to be included? If it's just the top-level directory list so to speak then we can probably add that at fairly low implementation cost since it's append-only and such...

[jethrogb]

I just need an index of all channel-rust-XYZ.toml files that exist, and I think that should be sufficient for must users. Note that XYZ may be a channel name (beta, nightly) or a version number (stable).

[Mark-Simulacrum]

Okay, I'll put that on my agenda for this week or so to investigate and see if we can provide that relatively easily, and if so start doing so.

[Mark-Simulacrum]

As a quick update I've internally setup weekly inventories of static.rust-lang.org and will check back in a couple days (AWS indicates we should get the first one in ~48 hours).

[andrewchambers]

You need tarballs so people packaging rust can download rust source code. The archives on github don't have the vendor directory. Please restore an index and working tarball links.

[Mark-Simulacrum]

Source tarballs, among others, are published for all releases to static.rust-lang.org.

https://manifests.rust-lang.org/manifests.txt is what I've come up with to hopefully satisfy @jethrogb's demands at least. Note that the format here is unstable, and we may stop updating this / remove it / etc at any time. It currently contains all nightly, beta, and stable manifests in the static.rust-lang.org/dist/{date}/ directories, which should be sufficient for essentially all use cases. Beyond that, let me know if additional information or a reworking of the format would be useful, by filing an issue at https://github.com/rust-lang/generate-manifest-list/.

Edit: Note that these results will be around a week out of date at all times, primarily for cost reasons, but this should hopefully not be a major concern for folks using this for bisection and such.

[jethrogb]

@jethrogb's demands

I like to think of them as requests 😄

I think the stable manifests are missing.

[andrewchambers]

@Mark-Simulacrum the link you posted is not valid.

I'll also note that I was unable to find full source links with the vendor dir.

[Mark-Simulacrum]

Ah, we've since decided we don't need a separate domain, so it should be semi-permanently at https://static.rust-lang.org/manifests.txt now.

I can't find any links to the source tarballs right now (we should add them to https://forge.rust-lang.org/infra/other-installation-methods.html, feel free to file a PR at https://github.com/rust-lang/rust-forge/); but they are published. For example, https://static.rust-lang.org/dist/2019-09-15/rustc-nightly-src.tar.gz or https://static.rust-lang.org/dist/2019-08-15/rustc-1.37.0-src.tar.xz.

[andrewchambers]

Note, those rustc tarballs don't have dependencies included, which seems to be a big oversight to me as there are references to tarballs with vendored dependencies scattered in the docs.

edit:
My mistake. They do have the contents. So we are just missing the links in a public place.