Wrong headers for some google-y crates from crates.io

[est31]

I am using reqwest to download crates from the Amazon CloudFront crates.io crate storage backend (to not artificially inflate the download counter). For most crates this works greatly, as in I can get files with the same check sum that is mentioned in the registry, but for some crates with google in their name, I stumbled across a bug: I got already extracted files in the tar format.

E.g. this URL works:

https://d19xqa3lc3clo8.cloudfront.net/crates/winapi/winapi-0.2.8.crate

but this URL does not:

https://d19xqa3lc3clo8.cloudfront.net/crates/google-adexchangeseller2/google-adexchangeseller2-0.1.2+20150309.crate

I've first thought that this is a reqwest bug as curl and wget seem to be able to download the files perfectly fine. But then I found out that Firefox has the same behaviour as reqwest. Just try to paste the two URLs into Firefox and inspect the downloaded files!

When you run curl -I on the two URLs above, you can see that the google crate has a Content-Encoding header while winapi lacks it.

I fortunately am able work around the issue by turning off gzip support in the ClientBuilder but the underlying issue is that these headers get sent by hosting.

This issue might probably re-surface if cargo decides to switch to using hyper.

[jtgeibel]

It looks like this is related to 53cbb26. It was fixed in June 2015, but the crate is "Last-Modified": "Wed, 25 Mar 2015 10:09:26 GMT".

[est31]

@jtgeibel that's interesting! In fact it might seem that the google- crates have been missed during some cleanup.

Version 0.1.15 of winapi was released in March 10, 2015. However, it has a Last-Modified: Thu, 09 Jul 2015 00:34:36 GMT header.

Version 0.1.14 has a Last-Modified: Thu, 09 Jul 2015 00:34:30 GMT header.

Both dates are very close to each other, as if someone ran a script to clean up crates.io.

[jtgeibel]

For future reference when cleaning these up on S3, my guess is that these were missed in the original cleanup script due to the build metadata (and + symbol) present in the file names.

[Turbo87]

@jtgeibel do you happen to remember if we saved this cleanup script anywhere? 😅

[Turbo87]

I tried to reproduce the issue but it looks like there is no Content-Encoding header anymore in both cases. if this ever comes back, feel free to reopen the issue with a more recent example :)