Make wasm32 buildbot test LLVM backend #42784

tlively · 2017-06-20T23:43:31Z

This adds the experimental targets option to configure so it can be used
by the builders and changes the wasm32 Dockerfile accordingly. Instead
of using LLVM from the emsdk, the builder's emscripten tools now uses
the Rust in-tree LLVM, since this is the one built with wasm support.

rust-highfive · 2017-06-20T23:43:36Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @aturon (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

tlively · 2017-06-20T23:49:36Z

r? @alexcrichton

cc @brson @eholk

The only thing this PR is missing to make the wasm32 builder fully functional is that clang (https://github.com/kripken/emscripten-fastcomp-clang/tree/incoming) needs to be in the src/llvm/tools directory so that emscripten can build the wasm libc bitcode files. It would be great if we could instead have a binary download to populate the emscripten cache or some other workaround, since having clang in-tree is undesirable.

Edit: I am exploring having the builder download a wasm-ready emscripten from the WebAssembly waterfall (https://wasm-stat.us/console) as a workaround.

tlively · 2017-06-20T23:51:31Z

src/ci/docker/disabled/wasm32/Dockerfile

@@ -20,7 +20,7 @@ RUN sh /scripts/dumb-init.sh
 # emscripten
 COPY scripts/emscripten.sh /scripts/
 RUN bash /scripts/emscripten.sh
-COPY wasm32/node.sh /usr/local/bin/node
+COPY disabled/wasm32/node.sh /usr/local/bin/node


This would have to be changed if the builder was ever moved out of the docker/disabled directory, but for now lets the builder be run locally without modification.

alexcrichton · 2017-06-21T15:52:21Z

src/tools/compiletest/src/runtest.rs

+        let mut env = self.props.rustc_env.clone();
+        // Tell emscripten to link using libc produced with LLVM backend
+        if self.config.target.contains("wasm32") && self.config.target.contains("experimental") {
+            env.push(("EMCC_WASM_BACKEND".to_string(), "1".to_string()));


Is this something that rustc should do automatically itself? (when compiling for the experimental wasm target)

Without this environment variable set, Emscripten tries to link the wasm bitcode file with the cached libc bitcode files for asmjs, which fails because asmjs and wasm have different memory layouts. Since this isn't just a debugging variable, rustc should definitely be setting it itself. When/if wasm32-experimental-emscripten is renamed to replace wasm32-unknown-emscripten, the part checking for "experimental" will need to be removed. It can't be removed now because then the current wasm32-unknown-emscripten would break.

Oh yeah sure having this is fine, it just seems like it should be in rustc's backend (e.g. the invocation of the linker) rather than only in compiletest.

Oh good point! I'll fix that for the next revision.

alexcrichton · 2017-06-21T15:53:05Z

src/ci/docker/scripts/emscripten.sh

 rm -f a.*

+# Make emscripten use Rust's LLVM


Mind expanding the comment here a bit? I'm actually not even sure what it means for emscripten to use someone else's llvm, isn't it precompiled?

tlively · 2017-06-22T01:21:28Z

src/ci/docker/disabled/wasm32/Dockerfile


-ENV RUST_CONFIGURE_ARGS --target=$TARGETS
+ENV RUST_CONFIGURE_ARGS --target=$TARGETS --experimental-targets=WebAssembly


This is problematic because as far as I know, rustc and LLVM aren't rebuilt for each bot. To actually get LLVM built with WebAssembly enabled will require setting this flag when it is built the first time. Is there a specific builder that is responsible for that initial build?

We rebuild LLVM and rustc in each bot. The LLVM build will possibly be problematic, since we normally cache it, so with this I'm not sure whether the cache will work properly for both the wasm32 and the non-wasm32 bots.

Oh this is fine, everything will work out with this configuration!

alexcrichton · 2017-06-22T14:04:25Z

@tlively at a high level can you explain to me why clang is needed? My current understanding of wasm is that the experimental backend in LLVM emites pure-wasm files (not bitcode) and then we use... something (emcc?) to link these together in the way that ld does for object files. Or does the experimental wasm backend still operate over bitcode?

tlively · 2017-06-22T17:56:38Z

@alexcrichton Unfortunately it's not quite as powerful as that. In the new experimental target, rustc still just emits bitcode files and tosses them over the fence to emcc. The difference is that it also tells emcc to use the experimental LLVM wasm backend, instead of going through fastcomp and asm.js like it otherwise would. In order for this to work 1) emscripten needs to be configured to use llvm tools built with the wasm backend enabled, and 2) emscripten needs to be able to build libc using the wasm backend or have it already cached so libc can be linked with the rustc-generated bitcode files. There are three ways to meet these requirements.

Build Rust's LLVM (and llvm tools) with the WebAssembly backend enabled, then configure emscripten to use those tools. Include clang in the tree so emscripten can use it as well. This is what I've been doing locally.
Build Rust's LLVM (and llvm tools) with the WebAssembly backend enabled, then configure emscripten to use those tools. Populate emscripten's cache by downloading prebuilt wasm libc bitcode files. I don't know where we could download prebuilt libc from, and this may still fail if emscripten looks for the existence of clang++ anyway.
Download wasm-ready emscripten and the llvm tools from the last known good build from the WebAssembly waterfall (https://wasm-stat.us/console). Probably no need for Rust's LLVM to be built with WebAssembly enabled, since its llvm tools will not be used. This is what I am trying to get working locally, but it's a little hairy.

alexcrichton · 2017-06-22T18:26:51Z

Ok thanks for the info! So it sounds like even though we're enabling the WebAssembly target in LLVM's backend, the rustc compiler itself isn't actually using it? As in, rustc is still emitting bitcode. I thought it was intended that we generate wasm "object files" with the wasm backend in LLVM, but is this not the case?

I guess though that emscripten then uses the wasm backend in LLVM to emit wasm files? (loads up bitcode and emits wasm files).

Where does this libc come from? Is it a bunch of C files that are compiled to wasm as well?

I guess what I'm sort of getting at is that this sounds like we're pretty close to not depending on wasm at all. If we leveraged LLVM's wasm back to generate wasm code, found some sort of linker for wasm code (maybe this is just LLVM?), and then got our "libc" written in Rust perhaps (or prebuilt), then we wouldn't need emcc at all?

I may be totally off base here though and not understanding. If you'd like I wouldn't mind chatting about this in a more high bandwidth setting (video/irc/etc), feel free to ping me on IRC if you'd like that!

eholk · 2017-06-22T19:15:33Z

@alexcrichton:

Ok thanks for the info! So it sounds like even though we're enabling the WebAssembly target in LLVM's backend, the rustc compiler itself isn't actually using it? As in, rustc is still emitting bitcode. I thought it was intended that we generate wasm "object files" with the wasm backend in LLVM, but is this not the case?

As far as I know, at the moment we don't have a great solution for linking wasm object files together. @sbc100 is making quick progress on this though, so hopefully we'll have something soon if we don't have it already.

alexcrichton · 2017-06-22T19:40:10Z

Good to hear @eholk!

For posterity @eholk, @tlively, and I chatted a bit on video about this, and the things I learned were:

The wasm32-unknown-emscripten target as implemented today generates asm.js, and then generates wasm from that asm.js
The wasm32-experimental-emscripten uses LLVM's native ability to emit wasm files.
Shelling out to emcc loads a bunch of bitcode into a giant LLVM module and then emits wasm from there.
The libc support library is lazily compiled by emcc, and requires clang/clang++
The rustc compiler, nor clang, require the experimental WebAssembly backend in LLVM. Only emcc which runs the LLVM tools in "wasm mode" requires the wasm backend.
The waterfall previously mentioned is a conglomeration of a bunch of wasm-realted tools. It's also generating a ton of artifacts from each successful build.

So it sounded to me like the wasm32-unknown-emscripten and wasm32-experimental-emscripten targets are quite similar today, but differ in rustc slightly based on how we call emcc and very differently in terms of the emcc compile-time requirements. The near term vision is then to get a travis builder for ourselves with these pieces:

We'll download emsdk as we're doing today, which includes clang but doesn't include a WebAssembly-backend-enabled LLVM
We'll build an LLVM with a WebAssembly backend for emcc to use
We'll arrange for the LLVM root we build to have clang (symlinked from the emsdk)
We'll instruct emcc to use the LLVM root so it uses WebAssembly-enabled tools.

In the longer term we'll:

Create a script to download tools from the waterfall mentioned above. These tools are the right clang/llvm/emcc/etc versions and have WebAssembly enabled as appropriate.
In our travis builder we'll download one of these waterfall artifacts as the "emsdk"
We'll still build our own LLVM, but we won't need to enable WebAssembly (for now)
We'll invoke emcc as usual

That "script" will ideally be usable by the community as well, and should hopefully enable easy usage of the native wasm backend of LLVM!

This adds the experimental targets option to configure so it can be used by the builders and changes the wasm32 Dockerfile accordingly. Instead of using LLVM from the emsdk, the builder's emscripten tools now uses the Rust in-tree LLVM, since this is the one built with wasm support.

This is used in wasm32-experimental-emscripten to ensure that emscripten links against the libc bitcode files produced by the wasm LLVM backend, instead of using fastcomp.

This modifies the builder to download and use the LLVM tools from the last known good build on the WebAssembly buildbot waterfall, since these tools are built with the WebAssembly LLVM backend enabled.

tlively · 2017-06-23T01:43:16Z

@alexcrichton This should be good to go! Instead of needing to symlink the wasm-ready LLVM tools, I only needed to change the emscripten configuration file, which should be relatively easy for end users to do. One advantage of this strategy over the previous one is that the builder automatically picks up the latest improvements to the wasm LLVM backend. There are tests that fail in WebAssembly instruction selection, but I have verified that these are real bugs, not bugs in the setup of the builder. I will let the right people know about these bugs and hopefully they will get fixed quickly.

alexcrichton · 2017-06-23T15:07:14Z

Looks good! It looks like one of the tests is failing though. We currently verify that we can roundtrip a Target through json so our custom target specs can take advantage of all the keys we have internally. You've added link_env here though which just means that we'll also need the associated serialize/deserialize functionality.

You can find deserialization here and serialization here. Perhaps this could be serialize/deserialized as a flat list of strings (with = signs) and then we parse that back out into key/value pairs?

eholk · 2017-06-23T17:33:49Z

FYI @vadimcn

Also turn WebAssembly backend back on in its builder.

alexcrichton · 2017-06-24T03:23:36Z

@bors: r+

Looks great!

bors · 2017-06-24T03:23:37Z

📌 Commit 4ad6a95 has been approved by alexcrichton

bors · 2017-06-24T07:04:21Z

⌛ Testing commit 4ad6a95 with merge 3a90ff1d8ebb94c6354072608bbf3a27751ab7dd...

bors · 2017-06-24T07:10:08Z

💔 Test failed - status-travis

Mark-Simulacrum · 2017-06-24T11:46:26Z

Looks like the asmjs failure is non-spurious:

[00:00:04] /scripts/emscripten.sh: line 37: jq: command not found

alexcrichton · 2017-06-24T18:51:36Z

@bors: r+

bors · 2017-06-24T18:51:37Z

📌 Commit c130b83 has been approved by alexcrichton

bors · 2017-06-24T22:34:14Z

⌛ Testing commit c130b83 with merge bc9822a...

Make wasm32 buildbot test LLVM backend This adds the experimental targets option to configure so it can be used by the builders and changes the wasm32 Dockerfile accordingly. Instead of using LLVM from the emsdk, the builder's emscripten tools now uses the Rust in-tree LLVM, since this is the one built with wasm support.

bors · 2017-06-25T01:01:17Z

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing bc9822a to master...

Split old and experimental wasm builders #42784 introduced configuration errors in the wasm builder by mixing different versions of the tools. This PR separates the wasm32-unknown-emscripten and wasm32-experimental-emscripten builders to resolve these errors.

rust-highfive assigned aturon Jun 20, 2017

rust-highfive assigned alexcrichton and unassigned aturon Jun 20, 2017

tlively commented Jun 20, 2017

View reviewed changes

tlively force-pushed the wasm-bot branch from d7f8f3b to fb68e81 Compare June 20, 2017 23:54

alexcrichton reviewed Jun 21, 2017

View reviewed changes

tlively commented Jun 22, 2017

View reviewed changes

alexcrichton added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jun 22, 2017

tlively added 3 commits June 22, 2017 18:34

Add target option for linker environment variables

16da303

This is used in wasm32-experimental-emscripten to ensure that emscripten links against the libc bitcode files produced by the wasm LLVM backend, instead of using fastcomp.

Add wasm32-experimental-emscripten to wasm builder

a4f9744

This modifies the builder to download and use the LLVM tools from the last known good build on the WebAssembly buildbot waterfall, since these tools are built with the WebAssembly LLVM backend enabled.

tlively force-pushed the wasm-bot branch from fb68e81 to a4f9744 Compare June 23, 2017 01:36

Add Target (de)serialization for environment vars

4ad6a95

Also turn WebAssembly backend back on in its builder.

Restore old emscripten.sh for use by asmjs

c130b83

bors mentioned this pull request Jun 25, 2017

[beta] backports #42841

Closed

bors merged commit c130b83 into rust-lang:master Jun 25, 2017

bors mentioned this pull request Jun 25, 2017

Refactor dockerfiles to have a common base image. #42691

Closed

tlively mentioned this pull request Jul 12, 2017

Split old and experimental wasm builders #43175

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make wasm32 buildbot test LLVM backend #42784

Make wasm32 buildbot test LLVM backend #42784

tlively commented Jun 20, 2017

rust-highfive commented Jun 20, 2017

tlively commented Jun 20, 2017 •

edited

Loading

tlively Jun 20, 2017

alexcrichton Jun 21, 2017

tlively Jun 21, 2017

alexcrichton Jun 21, 2017

tlively Jun 22, 2017

alexcrichton Jun 21, 2017

tlively Jun 22, 2017

Mark-Simulacrum Jun 22, 2017

alexcrichton Jun 22, 2017

alexcrichton commented Jun 22, 2017

tlively commented Jun 22, 2017

alexcrichton commented Jun 22, 2017

eholk commented Jun 22, 2017

alexcrichton commented Jun 22, 2017

tlively commented Jun 23, 2017

alexcrichton commented Jun 23, 2017

eholk commented Jun 23, 2017

alexcrichton commented Jun 24, 2017

bors commented Jun 24, 2017

bors commented Jun 24, 2017

bors commented Jun 24, 2017

Mark-Simulacrum commented Jun 24, 2017

alexcrichton commented Jun 24, 2017

bors commented Jun 24, 2017

bors commented Jun 24, 2017

bors commented Jun 25, 2017


		ENV RUST_CONFIGURE_ARGS --target=$TARGETS
		ENV RUST_CONFIGURE_ARGS --target=$TARGETS --experimental-targets=WebAssembly

Make wasm32 buildbot test LLVM backend #42784

Make wasm32 buildbot test LLVM backend #42784

Conversation

tlively commented Jun 20, 2017

rust-highfive commented Jun 20, 2017

tlively commented Jun 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Jun 22, 2017

tlively commented Jun 22, 2017

alexcrichton commented Jun 22, 2017

eholk commented Jun 22, 2017

alexcrichton commented Jun 22, 2017

tlively commented Jun 23, 2017

alexcrichton commented Jun 23, 2017

eholk commented Jun 23, 2017

alexcrichton commented Jun 24, 2017

bors commented Jun 24, 2017

bors commented Jun 24, 2017

bors commented Jun 24, 2017

Mark-Simulacrum commented Jun 24, 2017

alexcrichton commented Jun 24, 2017

bors commented Jun 24, 2017

bors commented Jun 24, 2017

bors commented Jun 25, 2017

tlively commented Jun 20, 2017 •

edited

Loading