Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a Docker container for doing automated builds for CloudABI. #47427

Merged
merged 4 commits into from
Jan 17, 2018

Conversation

EdSchouten
Copy link
Contributor

Setting up a cross compilation toolchain for CloudABI is relatively
easy. It's just a matter of installing a somewhat recent version of
Clang (5.0 preferred) and installing the corresponding
${target}-cxx-runtime package, containing a set of core C/C++ libraries
(libc, libc++, libunwind, etc).

Eventually it would be nice if we could also run x.py test. That,
however still requires some more work. Both libtest and compiletest
would need to be adjusted to deal with CloudABI's requirement of having
all of an application's dependencies injected. Let's settle for just
doing x.py dist for now.

Copy link
Member

@kennytm kennytm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that we are not going to add a new CI job in the short term, could you move the dist-cloudabi folder as src/ci/docker/disabled/dist-cloudabi?

.travis.yml Outdated
@@ -126,6 +126,8 @@ matrix:
if: branch = auto
- env: IMAGE=dist-armv7-linux DEPLOY=1
if: branch = auto
- env: IMAGE=dist-cloudabi DEPLOY=1
if: branch = auto
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want to put this as a separate job on the CI for now due to resource limitation.

@kennytm kennytm added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Jan 14, 2018
@EdSchouten
Copy link
Contributor Author

Aww... That's too bad. Be sure to get in touch with me if CPU capacity opens up.

@EdSchouten
Copy link
Contributor Author

Sorry if I'm sounding a bit negative, but not having automated builds for CloudABI is going to be more annoying than I had anticipated. If no automated builds can be provided, my idea would be to set up an automated build myself. That way, people can install Rust nightly on their operating system of choice and extend that with CloudABI support by extracting a rust-std-nightly-x86_64-unknown-cloudabi.tar.xz tarball hosted by me.

It turns out this won't work in practice, because rustc is quite picky:

error[E0514]: found crate `core` compiled by an incompatible version of rustc
  |
  = help: please recompile that crate using this compiler (rustc 1.25.0-nightly (e6072a7b3 2018-01-13))
  = note: the following crate versions were found:
          crate `core` compiled by rustc 1.25.0-dev (3f92e8d89 2018-01-14): /home/ed/.rustup/toolchains/nightly-x86_64-unknown-freebsd/lib/rustlib/x86_64-unknown-cloudabi/lib/libcore-14da3e8a93d4bc72.rlib

Do you by any chance know what would be the most elegant way to deal with this?

@Mark-Simulacrum
Copy link
Member

A potential solution is https://github.com/japaric/xargo. However, what you suggest would work, you'd just have to ship all of the "std" component that we normally do, including core. In theory, I could see rustup being extended to support using "components" from sources other than static.rust-lang.org, but I don't think there's a current maintainer of rustup to talk to about that.

@malbarbo
Copy link
Contributor

@EdSchouten You can try adding this to dist-various-1 or dist-various-2 builder.

@EdSchouten
Copy link
Contributor Author

@kennytm Integrating it in dist-various-* would be fine for me. Would that be all right for you?

@kennytm
Copy link
Member

kennytm commented Jan 16, 2018

@EdSchouten dist-various-* uses Ubuntu 16.04 LTS. Does this work for CloudABI? I'm not sure if we want to upgrade them to 17.10. cc @alexcrichton

@alexcrichton
Copy link
Member

@EdSchouten ah yes @Mark-Simulacrum and @malbarbo are both right in that if this is a separate builder we don't currently have the capacity to add more so it'll need to be disabled for now. If, however, it can be folded into an existing builder (which as it's just libstd I believe it should be totally feasible to do this) then our existing builders should have enough capacity to cross compile a new platform. We typically just pick whichever is fastest and throw it in there. Looks like the last successful run had dist-various-1 clocking in at 1h20m and dist-various-2 clocking in at 1h9m, so adding to dist-various-2 should do the trick!

Most of our containers use 16.04 as that was the latest when they were created, but I'm all for updating them as eagerly as possible (pending we're able to fix accidental regressions that creates of course). The dist-various-* containers should be able to get updated any time, the build will fail if they fail to build! Otherwise we in general don't want to have a dependency on a specific version, so if we are accidentally relying on something 16.04-specific we can fix that later.

In other words, let's see if we can fold this into dist-various-2? And you can feel free to update dist-various-2 if you'd like. You'll want to run locally though to make sure it all still works via ./src/ci/docker/run.sh dist-various-2

Setting up a cross compilation toolchain for CloudABI is relatively
easy. It's just a matter of installing a somewhat recent version of
Clang (5.0 preferred) and installing the corresponding
${target}-cxx-runtime package, containing a set of core C/C++ libraries
(libc, libc++, libunwind, etc).

Eventually it would be nice if we could also run 'x.py test'. That,
however still requires some more work. Both libtest and compiletest
would need to be adjusted to deal with CloudABI's requirement of having
all of an application's dependencies injected. Let's settle for just
doing 'x.py dist' for now.
There is not enough capacity to do automated builds for CloudABI at this
time.
As discussed in rust-lang#47427, let's not have a separate container for doing
CloudABI builds. It's a lot faster if we integrate it into an existing
container, so there's less duplication of what's being built.

Upgrade the existing container to Ubuntu 17.10, which is required for
CloudABI builds. The version of Clang shipped with 16.04 is not recent
enough to support CloudABI properly.
@EdSchouten
Copy link
Contributor Author

Hi @alexcrichton. Thanks for clarifying!

I just integrated CloudABI support into dist-various-2. It worked all right, but I did have to set DEPLOY=1. Apparently that's needed to work around #46781 / #43982. Let me know what you think!

@alexcrichton
Copy link
Member

Thanks @EdSchouten! I wonder if we could perhaps start with 1 or 2 targets just to make sure things work? We can always expand if we've still got capacity afterwards!

We'll turn on other architectures if it turns out we have enough
capacity.
@EdSchouten
Copy link
Contributor Author

Done! \o/

@alexcrichton
Copy link
Member

@bors: r+

@bors
Copy link
Contributor

bors commented Jan 16, 2018

📌 Commit dcf0cd0 has been approved by alexcrichton

@kennytm kennytm added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 16, 2018
@EdSchouten
Copy link
Contributor Author

@bors retry

Hmm... odd. The Travis page mentioned this build got cancelled? Not sure whether I have rights to force a retry.

@bors
Copy link
Contributor

bors commented Jan 17, 2018

@EdSchouten: 🔑 Insufficient privileges: and not in try users

@kennytm
Copy link
Member

kennytm commented Jan 17, 2018

@EdSchouten The build is canceled because Travis ded https://www.traviscistatus.com/incidents/c1s4dlyxd4lf

The build hasn't actually started, so retry does nothing (besides insufficient privilege).

kennytm added a commit to kennytm/rust that referenced this pull request Jan 17, 2018
Add a Docker container for doing automated builds for CloudABI.

Setting up a cross compilation toolchain for CloudABI is relatively
easy. It's just a matter of installing a somewhat recent version of
Clang (5.0 preferred) and installing the corresponding
`${target}-cxx-runtime` package, containing a set of core C/C++ libraries
(libc, libc++, libunwind, etc).

Eventually it would be nice if we could also run `x.py test`. That,
however still requires some more work. Both libtest and compiletest
would need to be adjusted to deal with CloudABI's requirement of having
all of an application's dependencies injected. Let's settle for just
doing `x.py dist` for now.
kennytm added a commit to kennytm/rust that referenced this pull request Jan 17, 2018
Add a Docker container for doing automated builds for CloudABI.

Setting up a cross compilation toolchain for CloudABI is relatively
easy. It's just a matter of installing a somewhat recent version of
Clang (5.0 preferred) and installing the corresponding
`${target}-cxx-runtime` package, containing a set of core C/C++ libraries
(libc, libc++, libunwind, etc).

Eventually it would be nice if we could also run `x.py test`. That,
however still requires some more work. Both libtest and compiletest
would need to be adjusted to deal with CloudABI's requirement of having
all of an application's dependencies injected. Let's settle for just
doing `x.py dist` for now.
bors added a commit that referenced this pull request Jan 17, 2018
@bors bors merged commit dcf0cd0 into rust-lang:master Jan 17, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants