mk: Hardcode the bootstrap key for each release #32731

alexcrichton · 2016-04-04T18:30:43Z

Starting with the 1.10.0 release we would like to bootstrap all compilers from
the previous stable release. For example the 1.10.0 compiler should bootstrap
from the literal 1.9.0 release artifacts. To do this, however, we need a way to
enable unstable features temporarily in a stable compiler (as the released
compiler is stable), but it turns out we already have a way to do that!

At compile time the configure script selects a CFG_BOOTSTRAP_KEY variable
value and then exports it into the makefiles. If the RUSTC_BOOTSTRAP_KEY
environment variable is set to this value, then the compiler is allowed to
"cheat" and use unstable features.

This method of choosing the bootstrap key, however, is problematic for the
intention of bootstrapping from the previous release. Each time a 1.9.0 compiler
is created, a new bootstrap key will be selected. That means that the 1.10.0
compiler will only compile from our literal release artifacts. Instead
distributions would like to bootstrap from their own compilers, so instead we
simply hardcode the bootstrap key for each release.

This patch uses the same CFG_FILENAME_EXTRA value (a hash of the release
string) as the bootstrap key. Consequently all 1.9.0 compilers, no matter where
they are compiled, will have the same bootstrap key. Additionally we won't need
to keep updating this as it'll be based on the release number anyway.

Once the 1.9.0 beta has been created, we can update the 1.10.0 nightly sources
(the master branch at that time) to bootstrap from that release using this
hard-coded bootstrap key. We will likely just hardcode into the makefiles what
the previous bootstrap key was and we'll change that whenever the stage0
compiler is updated.

Starting with the 1.10.0 release we would like to bootstrap all compilers from the previous stable release. For example the 1.10.0 compiler should bootstrap from the literal 1.9.0 release artifacts. To do this, however, we need a way to enable unstable features temporarily in a stable compiler (as the released compiler is stable), but it turns out we already have a way to do that! At compile time the configure script selects a `CFG_BOOTSTRAP_KEY` variable value and then exports it into the makefiles. If the `RUSTC_BOOTSTRAP_KEY` environment variable is set to this value, then the compiler is allowed to "cheat" and use unstable features. This method of choosing the bootstrap key, however, is problematic for the intention of bootstrapping from the previous release. Each time a 1.9.0 compiler is created, a new bootstrap key will be selected. That means that the 1.10.0 compiler will only compile from *our* literal release artifacts. Instead distributions would like to bootstrap from their own compilers, so instead we simply hardcode the bootstrap key for each release. This patch uses the same `CFG_FILENAME_EXTRA` value (a hash of the release string) as the bootstrap key. Consequently all 1.9.0 compilers, no matter where they are compiled, will have the same bootstrap key. Additionally we won't need to keep updating this as it'll be based on the release number anyway. Once the 1.9.0 beta has been created, we can update the 1.10.0 nightly sources (the `master` branch at that time) to bootstrap from that release using this hard-coded bootstrap key. We will likely just hardcode into the makefiles what the previous bootstrap key was and we'll change that whenever the stage0 compiler is updated.

rust-highfive · 2016-04-04T18:30:55Z

r? @brson

(rust_highfive has picked a reviewer for you, use r? to override)

alexcrichton · 2016-04-04T18:31:02Z

cc @rust-lang/core, @rust-lang/tools, @rust-lang/compiler

brson · 2016-04-04T18:38:46Z

@bors r+

bors · 2016-04-04T18:38:46Z

📌 Commit c822546 has been approved by brson

jnicholls · 2016-04-05T01:35:12Z

I'd love to hear why @eddyb is downing this. Is it because you're worried people will use the bootstrap key to cheat and use unstable features in a stable compiler?

eddyb · 2016-04-05T01:39:43Z

@jnicholls It has been possible since 1.0 (release build logs are public) and everyone involved knows my position (compile different "rustc drivers" for the different uscases - this would be even easier with rustbuild), but I agree that distro maintainers would see this as pointless complexity so I'll just leave it at "I don't like this but not enough to actually to react strongly to it (again)".

…=brson mk: Hardcode the bootstrap key for each release Starting with the 1.10.0 release we would like to bootstrap all compilers from the previous stable release. For example the 1.10.0 compiler should bootstrap from the literal 1.9.0 release artifacts. To do this, however, we need a way to enable unstable features temporarily in a stable compiler (as the released compiler is stable), but it turns out we already have a way to do that! At compile time the configure script selects a `CFG_BOOTSTRAP_KEY` variable value and then exports it into the makefiles. If the `RUSTC_BOOTSTRAP_KEY` environment variable is set to this value, then the compiler is allowed to "cheat" and use unstable features. This method of choosing the bootstrap key, however, is problematic for the intention of bootstrapping from the previous release. Each time a 1.9.0 compiler is created, a new bootstrap key will be selected. That means that the 1.10.0 compiler will only compile from *our* literal release artifacts. Instead distributions would like to bootstrap from their own compilers, so instead we simply hardcode the bootstrap key for each release. This patch uses the same `CFG_FILENAME_EXTRA` value (a hash of the release string) as the bootstrap key. Consequently all 1.9.0 compilers, no matter where they are compiled, will have the same bootstrap key. Additionally we won't need to keep updating this as it'll be based on the release number anyway. Once the 1.9.0 beta has been created, we can update the 1.10.0 nightly sources (the `master` branch at that time) to bootstrap from that release using this hard-coded bootstrap key. We will likely just hardcode into the makefiles what the previous bootstrap key was and we'll change that whenever the stage0 compiler is updated.

Rollup of 11 pull requests - Successful merges: #32016, #32583, #32699, #32729, #32731, #32738, #32741, #32745, #32748, #32757, #32786 - Failed merges: #32773

…=brson mk: Hardcode the bootstrap key for each release Starting with the 1.10.0 release we would like to bootstrap all compilers from the previous stable release. For example the 1.10.0 compiler should bootstrap from the literal 1.9.0 release artifacts. To do this, however, we need a way to enable unstable features temporarily in a stable compiler (as the released compiler is stable), but it turns out we already have a way to do that! At compile time the configure script selects a `CFG_BOOTSTRAP_KEY` variable value and then exports it into the makefiles. If the `RUSTC_BOOTSTRAP_KEY` environment variable is set to this value, then the compiler is allowed to "cheat" and use unstable features. This method of choosing the bootstrap key, however, is problematic for the intention of bootstrapping from the previous release. Each time a 1.9.0 compiler is created, a new bootstrap key will be selected. That means that the 1.10.0 compiler will only compile from *our* literal release artifacts. Instead distributions would like to bootstrap from their own compilers, so instead we simply hardcode the bootstrap key for each release. This patch uses the same `CFG_FILENAME_EXTRA` value (a hash of the release string) as the bootstrap key. Consequently all 1.9.0 compilers, no matter where they are compiled, will have the same bootstrap key. Additionally we won't need to keep updating this as it'll be based on the release number anyway. Once the 1.9.0 beta has been created, we can update the 1.10.0 nightly sources (the `master` branch at that time) to bootstrap from that release using this hard-coded bootstrap key. We will likely just hardcode into the makefiles what the previous bootstrap key was and we'll change that whenever the stage0 compiler is updated.

Rollup of 7 pull requests - Successful merges: #32687, #32729, #32731, #32732, #32734, #32737, #32741 - Failed merges:

Since PR rust-lang#32731 we’re hardcoding the BOOTSTRAP_KEY to a well known value, so there’s no reason to have any key at all. Lets just use a regular environment variable instead. Hooray for simpler code.

This commit removes all infrastructure from the repository for our so-called snapshots to instead bootstrap the compiler from stable releases. Bootstrapping from a previously stable release is a long-desired feature of distros because they're not fans of downloading binary stage0 blobs from us. Additionally, this makes our own CI easier as we can decommission all of the snapshot builders and start having a regular cadence to when we update the stage0 compiler. A new `src/etc/get-stage0.py` script was added which shares some code with `src/bootstrap/bootstrap.py` to read a new file, `src/stage0.txt`, which lists the current stage0 compiler as well as cargo that we bootstrap from. This script will download the relevant `rustc` package an unpack it into `$target/stage0` as we do today. One problem of bootstrapping from stable releases is that we're not able to compile unstable code (e.g. all the `#![feature]` directives in libcore/libstd). To overcome this we employ two strategies: * The bootstrap key of the previous compiler is hardcoded into `src/stage0.txt` (enabled as a result of rust-lang#32731) and exported by the build system. This enables nightly features in the compiler we download. * The standard library and compiler are pinned to a specific stage0, which doesn't change, so we're guaranteed that we'll continue compiling as we start from a known fixed source. The process for making a release will also need to be tweaked now to continue to cadence of bootstrapping from the previous release. This process looks like: 1. Merge `beta` to `stable` 2. Produce a new stable compiler. 3. Change `master` to bootstrap from this new stable compiler. 4. Merge `master` to `beta` 5. Produce a new beta compiler 6. Change `master` to bootstrap from this new beta compiler. Step 3 above should involve very few changes as `master` was previously bootstrapping from `beta` which is the same as `stable` at that point in time. Step 6, however, is where we benefit from removing lots of `#[cfg(stage0)]` and get to use new features. This also shouldn't slow the release too much as steps 1-5 requires little work other than waiting and step 6 just needs to happen at some point during a release cycle, it's not time sensitive. Closes rust-lang#29555 Closes rust-lang#29557

mk: Bootstrap from stable instead of snapshots This commit removes all infrastructure from the repository for our so-called snapshots to instead bootstrap the compiler from stable releases. Bootstrapping from a previously stable release is a long-desired feature of distros because they're not fans of downloading binary stage0 blobs from us. Additionally, this makes our own CI easier as we can decommission all of the snapshot builders and start having a regular cadence to when we update the stage0 compiler. A new `src/etc/get-stage0.py` script was added which shares some code with `src/bootstrap/bootstrap.py` to read a new file, `src/stage0.txt`, which lists the current stage0 compiler as well as cargo that we bootstrap from. This script will download the relevant `rustc` package an unpack it into `$target/stage0` as we do today. One problem of bootstrapping from stable releases is that we're not able to compile unstable code (e.g. all the `#![feature]` directives in libcore/libstd). To overcome this we employ two strategies: * The bootstrap key of the previous compiler is hardcoded into `src/stage0.txt` (enabled as a result of #32731) and exported by the build system. This enables nightly features in the compiler we download. * The standard library and compiler are pinned to a specific stage0, which doesn't change, so we're guaranteed that we'll continue compiling as we start from a known fixed source. The process for making a release will also need to be tweaked now to continue to cadence of bootstrapping from the previous release. This process looks like: 1. Merge `beta` to `stable` 2. Produce a new stable compiler. 3. Change `master` to bootstrap from this new stable compiler. 4. Merge `master` to `beta` 5. Produce a new beta compiler 6. Change `master` to bootstrap from this new beta compiler. Step 3 above should involve very few changes as `master` was previously bootstrapping from `beta` which is the same as `stable` at that point in time. Step 6, however, is where we benefit from removing lots of `#[cfg(stage0)]` and get to use new features. This also shouldn't slow the release too much as steps 1-5 requires little work other than waiting and step 6 just needs to happen at some point during a release cycle, it's not time sensitive. Closes #29555 Closes #29557

rust-highfive assigned brson Apr 4, 2016

Manishearth mentioned this pull request Apr 6, 2016

Rollup of 13 pull requests #32759

Closed

Manishearth mentioned this pull request Apr 7, 2016

Rollup of 11 pull requests #32787

Closed

bors added a commit that referenced this pull request Apr 7, 2016

Auto merge of #32787 - Manishearth:rollup, r=Manishearth

6a55f4f

Rollup of 11 pull requests - Successful merges: #32016, #32583, #32699, #32729, #32731, #32738, #32741, #32745, #32748, #32757, #32786 - Failed merges: #32773

Manishearth mentioned this pull request Apr 7, 2016

Rollup of 7 pull requests #32800

Merged

bors added a commit that referenced this pull request Apr 7, 2016

Auto merge of #32800 - Manishearth:rollup, r=Manishearth

dde35e7

Rollup of 7 pull requests - Successful merges: #32687, #32729, #32731, #32732, #32734, #32737, #32741 - Failed merges:

nagisa mentioned this pull request Apr 8, 2016

mk: Simplify boostrap key strategy #32812

Closed

bors merged commit c822546 into rust-lang:master Apr 8, 2016

alexcrichton mentioned this pull request Apr 13, 2016

Feature/Stability staging is flaky #21230

Closed

alexcrichton mentioned this pull request Apr 13, 2016

mk: Bootstrap from stable instead of snapshots #32942

Merged

alexcrichton deleted the known-bootstrap-key branch April 14, 2016 00:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mk: Hardcode the bootstrap key for each release #32731

mk: Hardcode the bootstrap key for each release #32731

alexcrichton commented Apr 4, 2016

rust-highfive commented Apr 4, 2016

alexcrichton commented Apr 4, 2016

brson commented Apr 4, 2016

bors commented Apr 4, 2016

jnicholls commented Apr 5, 2016

eddyb commented Apr 5, 2016

mk: Hardcode the bootstrap key for each release #32731

mk: Hardcode the bootstrap key for each release #32731

Conversation

alexcrichton commented Apr 4, 2016

rust-highfive commented Apr 4, 2016

alexcrichton commented Apr 4, 2016

brson commented Apr 4, 2016

bors commented Apr 4, 2016

jnicholls commented Apr 5, 2016

eddyb commented Apr 5, 2016