Some crate files in the index can't be downloaded

[est31]

Over the past 2 days I have been downloading the crates from crates.io using a tool I have written for that purpose and I've discovered that some crates, while mentioned in the index, can't be downloaded from the backend.

The affected (crate, version) pairs:

cargo-ctags v0.2.3
--
glib-2-0-sys v0.0.1
glib-2-0-sys v0.0.2
glib-2-0-sys v0.0.3
glib-2-0-sys v0.0.4
glib-2-0-sys v0.0.5
glib-2-0-sys v0.0.6
glib-2-0-sys v0.0.7
glib-2-0-sys v0.0.8
glib-2-0-sys v0.1.0
glib-2-0-sys v0.1.1
glib-2-0-sys v0.1.2
glib-2-0-sys v0.2.0
--
gobject-2-0-sys v0.0.2
gobject-2-0-sys v0.0.3
gobject-2-0-sys v0.0.4
gobject-2-0-sys v0.0.5
gobject-2-0-sys v0.0.6
gobject-2-0-sys v0.0.7
gobject-2-0-sys v0.0.8
gobject-2-0-sys v0.0.9
gobject-2-0-sys v0.1.0
gobject-2-0-sys v0.2.0

This is exposed to normal users of crates.io as well. If you do the following in your Cargo.toml:

[dependencies]
glib-2-0-sys = "0.0.1"

You will get this error:

$ cargo build
    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading glib-2-0-sys v0.0.1                                                
error: unable to get packages from source                                       

Caused by:
  failed to get 200 response from `https://d19xqa3lc3clo8.cloudfront.net/crates/glib-2-0-sys/glib-2-0-sys-0.0.1.crate`, got 403

Looking at the git logs of the index it seems to me that deletions always mean that the specific versions are also deleted from the index. These versions are still present.

[progval]

ping?

[ashleygwilliams]

here's a more complete list posted by iscgar on IRC

• through crates.io redirect: https://pastebin.com/raw/4ZspDnZv
• directly hitting bucket: https://pastebin.com/WqbDV1P2

[est31]

@ashleygwilliams that list contains packages with uppercase letters in their name, like RustyXML, Lattice or FPS. And those packages can be downloaded, you just need to use the properly cased name. E.g. for RustyXML version 0.1.0 you can use these URLs:

• https://d19xqa3lc3clo8.cloudfront.net/crates/RustyXML/rustyxml-0.1.0.crate
• https://static.crates.io/crates/RustyXML/RustyXML-0.1.0.crate
• https://crates.io/api/v1/crates/RustyXML/0.1.0/download

Most likely iscgar's downloader tried to use all lowercase crate names. I've been there myself :p.

As for why glib-2-0-sys is different:

The casing for RustyXML is observable from the json file in the index. Similarly for Lattice and FPS. On the other hand, glib-2-0-sys has the lowercase variant in it's json file. Maybe there is a casing pattern for glib-2-0-sys that works, I haven't tried any. But even if there were, there is clearly a bug here in the json file for glib-2-0-sys. This would then also be a cause for why glib-2-0-sys is unusable from Cargo.toml. cargo probably uses these json files.

[est31]

Hmmm this commit looks very suspicious: rust-lang/crates.io-index@610b81f

So it seems that it has to be glib-2_0-sys. This URL does work and it also has the same hash as the corresponding entry in the json file, f915436b932305fa0dddf926ce1bb513b5a099e78c5223424027cad256c5a6fe:

https://static.crates.io/crates/glib-2_0-sys/glib-2_0-sys-0.0.1.crate

So apparently the rename was performed by @alexcrichton on the index only, but not on the actual files in S3.

This rename explains glib-2-0-sys as well as gobject-2-0-sys but doesn't explain cargo-ctags. I see no human commit in the index for that crate: https://github.com/rust-lang/crates.io-index/commits/master/ca/rg/cargo-ctags

[iscgar]

@est31 right, my bad. The list of 403s is indeed now down to cargo-ctags-0.2.3.

[iscgar]

cargo-ctags-0.2.3 seems to be missing from the crates.io database (it's the only version of cargo-ctags that is missing)

[est31]

@iscgar No problem!

[iscgar]

The cargo-ctags-0.2.3 situation seems to be the same as https://github.com/rust-lang/crates.io-index/pull/11 (see #1447) so the fix here should probably be the same: just remove the 0.2.3 line from the index.

[sgrif]

cargo-ctags 0.2.3 was likely caused by a rare scenario that we've seen a few times where the index gets successfully updated, but something happens (server restart, database issue, etc) before the database transaction was committed. I've checked the index against our database and removed all cases of any files/versions in the index that aren't present in our database in rust-lang/crates.io-index@e58b304

If any additional cases arise in the future, please open a new issue.