rustup update nightly sometimes doesn't see the most recent tarball

[nox]

Today, rustup update nightly gives the same version as nightly-2016-11-21 even though nightly-2016-11-22 exists.

[TimNN]

This seems like a caching Problem:

Using Chrome to download, https://static.rust-lang.org/dist/channel-rust-nightly.toml contains sections like

[pkg.rust.target.x86_64-apple-darwin]
available = true
url = "https://static.rust-lang.org/dist/2016-11-22/rust-nightly-x86_64-apple-darwin.tar.gz"
hash = "e969929b629d5519525be3c43e858d64c8696b35d8f899a6af89a93897d327e2"

Whereas using curl gives

[pkg.rust.target.x86_64-apple-darwin]
available = true
url = "https://static.rust-lang.org/dist/2016-11-21/rust-nightly-x86_64-apple-darwin.tar.gz"
hash = "1ef354dabd1747601ae0343e345186d6e1cd23f6d7376ad9b464b898aba861a7"

Adding -H 'accept-encoding: gzip' --compressed to the curl command produces the correct nightly.

[TimNN]

From the Cloudfront docs at http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html, it seems that the compressed / non-compressed caches are separate:

5. If the compressed file is not in the cache, CloudFront forwards the request to the origin server, which can be either an Amazon S3 bucket or a custom origin.

   Note: If CloudFront has an uncompressed version of the file in the cache, it still forwards a request to the origin.

This sounds to me like this might be a bug with CloudFront + S3 + CF Compression + CF Invalidation

[TimNN]

By the way, here are the response headers:

$ curl 'https://static.rust-lang.org/dist/channel-rust-nightly.toml' --verbose -H 'accept-encoding: gzip'> /dev/null
< HTTP/1.1 200 OK
< Content-Type: binary/octet-stream
< Content-Length: 145808
< Connection: keep-alive
< Date: Wed, 23 Nov 2016 10:54:12 GMT
< Last-Modified: Tue, 22 Nov 2016 18:58:55 GMT
< ETag: "077d48bbe58542c462bd74dc68c7ac46"
< Accept-Ranges: bytes
< Server: AmazonS3
< Age: 1951
< X-Cache: Hit from cloudfront
< Via: 1.1 a71703bb7880a52614551015740610ad.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: y4JXrxYAYG5dyL-3ZyMK5Y3cdPn7pFo6kP5JXPEX0fPPyQ4A_LaIOQ==

$ curl 'https://static.rust-lang.org/dist/channel-rust-nightly.toml' --verbose > /dev/null
< HTTP/1.1 200 OK
< Content-Type: binary/octet-stream
< Content-Length: 145808
< Connection: keep-alive
< Date: Tue, 22 Nov 2016 15:20:19 GMT
< Last-Modified: Mon, 21 Nov 2016 18:46:43 GMT
< ETag: "10b5e1f2d7fb3a7ee975612aeae7e414"
< Accept-Ranges: bytes
< Server: AmazonS3
< Age: 72319
< X-Cache: Hit from cloudfront
< Via: 1.1 13be6f06029966b75af2e6bcb4d1867a.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: LgqOhcD4PbSHDjciMco0y0ORqWYuqHIqVGdh-WmCKXrpkSPut7FLhg==

[alexcrichton]

This is likely a dupe of rust-lang/rust#21239

[TimNN]

@alexcrichton: Ah, I though there was already an invalidation step on the nightly builders, which just wasn't working correctly.

[nox]

Oh I see.

[jonhoo]

This seems to be happening again @alexcrichton?

$ curl -s https://dev-static.rust-lang.org/dist/channel-rust-nightly.toml | grep -B1 version
date = "2017-05-07"
manifest-version = "2"
--
[pkg.cargo]
version = "0.19.0-nightly (fa7584c14 2017-04-26)"
--
[pkg.rls]
version = "0.1.0-nightly (67babd2d6 2017-04-27)"
--
[pkg.rust]
version = "1.19.0-nightly (5b31bf851 2017-05-07)"
--
[pkg.rust-analysis]
version = "1.19.0-nightly (5b31bf851 2017-05-07)"
--
[pkg.rust-docs]
version = "1.19.0-nightly (5b31bf851 2017-05-07)"
--
[pkg.rust-mingw]
version = "1.19.0-nightly (5b31bf851 2017-05-07)"
--
[pkg.rust-src]
version = "1.19.0-nightly (5b31bf851 2017-05-07)"
--
[pkg.rust-std]
version = "1.19.0-nightly (5b31bf851 2017-05-07)"
--
[pkg.rustc]
version = "1.19.0-nightly (5b31bf851 2017-05-07)"
$ rustup update
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: syncing channel updates for 'beta-x86_64-unknown-linux-gnu'
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'

   stable-x86_64-unknown-linux-gnu unchanged - rustc 1.17.0 (56124baa9 2017-04-24)
     beta-x86_64-unknown-linux-gnu unchanged - rustc 1.18.0-beta.1 (4dce67253 2017-04-25)
  nightly-x86_64-unknown-linux-gnu unchanged - rustc 1.19.0-nightly (f4209651e 2017-05-05)

[jonhoo]

Ahhh, never mind, I was looking at dev-static, not static.