Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rustbuild: Add support for compiling Cargo #39917

Merged
merged 2 commits into from
Mar 4, 2017

Conversation

alexcrichton
Copy link
Member

This commit adds support to rustbuild for compiling Cargo as part of the release
process. Previously rustbuild would simply download a Cargo snapshot and
repackage it. With this change we should be able to turn off artifacts from the
rust-lang/cargo repository and purely rely on the artifacts Cargo produces here.

The infrastructure added here is intended to be extensible to other components,
such as the RLS. It won't exactly be a one-line addition, but the addition of
Cargo didn't require too much hooplah anyway.

The process for release Cargo will now look like:

  • The rust-lang/rust repository has a Cargo submodule which is used to build a
    Cargo to pair with the rust-lang/rust release
  • Periodically we'll update the cargo submodule as necessary on rust-lang/rust's
    master branch
  • When branching beta we'll create a new branch of Cargo (as we do today), and
    the first commit to the beta branch will be to update the Cargo submodule to
    this exact revision.
  • When branching stable, we'll ensure that the Cargo submodule is updated and
    then make a stable release.

Backports to Cargo will look like:

  • Send a PR to cargo's master branch
  • Send a PR to cargo's release branch (e.g. rust-1.16.0)
  • Send a PR to rust-lang/rust's beta branch updating the submodule
  • Eventually send a PR to rust-lang/rust's master branch updating the submodule

For reference, the process to add a new component to the rust-lang/rust release
would look like:

  • Add $foo as a submodule in src/tools
  • Add a tool-$foo step which compiles $foo with the specified compiler,
    likely mirroring what Cargo does.
  • Add a dist-$foo step which uses src/tools/$foo and the tool-$foo output
    to create a rust-installer package for $foo likely mirroring what Cargo
    does.
  • Update the dist-extended step with a new dependency on dist-$foo
  • Update src/tools/build-manifest for the new component.

@rust-highfive
Copy link
Collaborator

r? @brson

(rust_highfive has picked a reviewer for you, use r? to override)

@bors
Copy link
Contributor

bors commented Feb 20, 2017

☔ The latest upstream changes (presumably #39966) made this pull request unmergeable. Please resolve the merge conflicts.

@@ -1,5 +1,5 @@
[package]
name = "cargotest"
name = "cargotest2"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this renamed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh this conflicted with the cargotest package in Cargo itself, so I figured it'd be easiest to rename this here

@brson
Copy link
Contributor

brson commented Feb 22, 2017

@bors r+

All the openssl logic in bootstrap is 😢

@bors
Copy link
Contributor

bors commented Feb 22, 2017

📌 Commit 3120181 has been approved by brson

@alexcrichton
Copy link
Member Author

@bors: r=brson

@bors
Copy link
Contributor

bors commented Feb 22, 2017

📌 Commit ed2f0d6 has been approved by brson

@alexcrichton
Copy link
Member Author

FWIW rust-openssl may get support soon for building OpenSSL given a source tarball, so we could at least do away with all the grungy build logic.

};

configure.arg(format!("--prefix={}", dst.display()));
configure.arg("no-dso");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be worth adding no-ssl2 no-ssl3 no-comp additionally just to be safe.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@alexcrichton
Copy link
Member Author

@bors: r=brson

@bors
Copy link
Contributor

bors commented Feb 22, 2017

📌 Commit b8fc75f has been approved by brson

@alexcrichton
Copy link
Member Author

@bors: r=brson

@bors
Copy link
Contributor

bors commented Feb 23, 2017

📌 Commit 28ea763 has been approved by brson

@bors
Copy link
Contributor

bors commented Feb 24, 2017

☔ The latest upstream changes (presumably #39851) made this pull request unmergeable. Please resolve the merge conflicts.

@bors
Copy link
Contributor

bors commented Feb 25, 2017

🔒 Merge conflict

@alexcrichton
Copy link
Member Author

@bors: r=brson

@bors
Copy link
Contributor

bors commented Feb 25, 2017

📌 Commit be5cb13 has been approved by brson

@bors
Copy link
Contributor

bors commented Feb 25, 2017

⌛ Testing commit be5cb13 with merge 9d6ed79...

@bors
Copy link
Contributor

bors commented Feb 25, 2017

💔 Test failed - status-travis

@alexcrichton
Copy link
Member Author

@bors: r=brson

This commit adds support to rustbuild for compiling Cargo as part of the release
process. Previously rustbuild would simply download a Cargo snapshot and
repackage it. With this change we should be able to turn off artifacts from the
rust-lang/cargo repository and purely rely on the artifacts Cargo produces here.

The infrastructure added here is intended to be extensible to other components,
such as the RLS. It won't exactly be a one-line addition, but the addition of
Cargo didn't require too much hooplah anyway.

The process for release Cargo will now look like:

* The rust-lang/rust repository has a Cargo submodule which is used to build a
  Cargo to pair with the rust-lang/rust release
* Periodically we'll update the cargo submodule as necessary on rust-lang/rust's
  master branch
* When branching beta we'll create a new branch of Cargo (as we do today), and
  the first commit to the beta branch will be to update the Cargo submodule to
  this exact revision.
* When branching stable, we'll ensure that the Cargo submodule is updated and
  then make a stable release.

Backports to Cargo will look like:

* Send a PR to cargo's master branch
* Send a PR to cargo's release branch (e.g. rust-1.16.0)
* Send a PR to rust-lang/rust's beta branch updating the submodule
* Eventually send a PR to rust-lang/rust's master branch updating the submodule

For reference, the process to add a new component to the rust-lang/rust release
would look like:

* Add `$foo` as a submodule in `src/tools`
* Add a `tool-$foo` step which compiles `$foo` with the specified compiler,
  likely mirroring what Cargo does.
* Add a `dist-$foo` step which uses `src/tools/$foo` and the `tool-$foo` output
  to create a rust-installer package for `$foo` likely mirroring what Cargo
  does.
* Update the `dist-extended` step with a new dependency on `dist-$foo`
* Update `src/tools/build-manifest` for the new component.
@alexcrichton
Copy link
Member Author

@bors: r=brson

Why thank you @bors, I did indeed not intend to check in 100MB build trees into the repo

@bors
Copy link
Contributor

bors commented Mar 3, 2017

📌 Commit 44a01b8 has been approved by brson

@bors
Copy link
Contributor

bors commented Mar 3, 2017

⌛ Testing commit 44a01b8 with merge 7d2bdb0...

@bors
Copy link
Contributor

bors commented Mar 3, 2017

💔 Test failed - status-travis

@alexcrichton
Copy link
Member Author

@bors: retry

@bors
Copy link
Contributor

bors commented Mar 3, 2017

⌛ Testing commit 44a01b8 with merge 4e0cfbd...

@bors
Copy link
Contributor

bors commented Mar 3, 2017

💔 Test failed - status-travis

@alexcrichton
Copy link
Member Author

alexcrichton commented Mar 3, 2017 via email

@bors
Copy link
Contributor

bors commented Mar 3, 2017

⌛ Testing commit 44a01b8 with merge 5208090...

bors added a commit that referenced this pull request Mar 3, 2017
rustbuild: Add support for compiling Cargo

This commit adds support to rustbuild for compiling Cargo as part of the release
process. Previously rustbuild would simply download a Cargo snapshot and
repackage it. With this change we should be able to turn off artifacts from the
rust-lang/cargo repository and purely rely on the artifacts Cargo produces here.

The infrastructure added here is intended to be extensible to other components,
such as the RLS. It won't exactly be a one-line addition, but the addition of
Cargo didn't require too much hooplah anyway.

The process for release Cargo will now look like:

* The rust-lang/rust repository has a Cargo submodule which is used to build a
  Cargo to pair with the rust-lang/rust release
* Periodically we'll update the cargo submodule as necessary on rust-lang/rust's
  master branch
* When branching beta we'll create a new branch of Cargo (as we do today), and
  the first commit to the beta branch will be to update the Cargo submodule to
  this exact revision.
* When branching stable, we'll ensure that the Cargo submodule is updated and
  then make a stable release.

Backports to Cargo will look like:

* Send a PR to cargo's master branch
* Send a PR to cargo's release branch (e.g. rust-1.16.0)
* Send a PR to rust-lang/rust's beta branch updating the submodule
* Eventually send a PR to rust-lang/rust's master branch updating the submodule

For reference, the process to add a new component to the rust-lang/rust release
would look like:

* Add `$foo` as a submodule in `src/tools`
* Add a `tool-$foo` step which compiles `$foo` with the specified compiler,
  likely mirroring what Cargo does.
* Add a `dist-$foo` step which uses `src/tools/$foo` and the `tool-$foo` output
  to create a rust-installer package for `$foo` likely mirroring what Cargo
  does.
* Update the `dist-extended` step with a new dependency on `dist-$foo`
* Update `src/tools/build-manifest` for the new component.
@bors
Copy link
Contributor

bors commented Mar 4, 2017

☀️ Test successful - status-appveyor, status-travis
Approved by: brson
Pushing 5208090 to master...

@shepmaster
Copy link
Member

@alexcrichton after pulling this change, I can no longer build:

$ rm -rf build-release/
$ mkdir build-release
$ cd build-release/
$ ../configure
$ make
downloading https://static.rust-lang.org/dist/2017-02-01/rust-std-beta-x86_64-apple-darwin.tar.gz
######################################################################## 100.0%
extracting /Users/shep/Projects/rust/build-release/build/cache/2017-02-01/rust-std-beta-x86_64-apple-darwin.tar.gz
downloading https://static.rust-lang.org/dist/2017-02-01/rustc-beta-x86_64-apple-darwin.tar.gz
######################################################################## 100.0%
extracting /Users/shep/Projects/rust/build-release/build/cache/2017-02-01/rustc-beta-x86_64-apple-darwin.tar.gz
downloading https://s3.amazonaws.com/rust-lang-ci/cargo-builds/bfee18f73287687c543bda8c35e4e33808792715/cargo-nightly-x86_64-apple-darwin.tar.gz
######################################################################## 100.0%
extracting /Users/shep/Projects/rust/build-release/build/cache/bfee18f73287687c543bda8c35e4e33808792715/cargo-nightly-x86_64-apple-darwin.tar.gz
error: failed to read `/Users/shep/Projects/rust/src/tools/cargo/Cargo.toml`

Caused by:
  No such file or directory (os error 2)
make: *** [all] Error 101

Is there another step I'm unaware of?

@alexcrichton
Copy link
Member Author

Ah yup that'd do it.

Cargo's a submodule. We'll have to move submodule updates sooner in the process or warn about it. You can fix this with:

git submodule update --init

@alexcrichton alexcrichton deleted the build-cargo branch March 4, 2017 04:38
@Mark-Simulacrum
Copy link
Member

Should that perhaps be posted in a more visible location (e.g., internals)? I suspect pretty much all developers working on rustc will run into it.

@shepmaster
Copy link
Member

all developers working on rustc

I think everyone is playing that new Zelda game today 😉

More seriously, I just assumed that it would be fixed soon enough that not many people would run into it. I'm also a bit surprised that the build machines didn't have any issues. I guess there's differences between what a human does and the autobuilds?

@alexcrichton
Copy link
Member Author

I've sent a PR to fix the issue @shepmaster ran into.

Sorry for the breakage!

Also yeah the auto builds explicitly update submodules for... reasons. So that's why they didn't run into this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.