rustc: Split Emscripten to a separate codegen backend #47730

alexcrichton · 2018-01-25T04:25:08Z

This commit introduces a separately compiled backend for Emscripten, avoiding
compiling the JSBackend target in the main LLVM codegen backend. This builds
on the foundation provided by #47671 to create a new codegen backend dedicated
solely to Emscripten, removing the JSBackend of the main codegen backend in
the process.

A new field was added to each target for this commit which specifies the backend
to use for translation, the default being llvm which is the main backend that
we use. The Emscripten targets specify an emscripten backend instead of the
main llvm one.

There's a whole bunch of consequences of this change, but I'll try to enumerate
them here:

A second LLVM submodule was added in this commit. The main LLVM submodule
will soon start to drift from the Emscripten submodule, but currently they're
both at the same revision.
Logic was added to rustbuild to not build the Emscripten backend by default.
This is gated behind a --enable-emscripten flag to the configure script. By
default users should neither check out the emscripten submodule nor compile
it.
The init_repo.sh script was updated to fetch the Emscripten submodule from
GitHub the same way we do the main LLVM submodule (a tarball fetch).
The Emscripten backend, turned off by default, is still turned on for a number
of targets on CI. We'll only be shipping an Emscripten backend with Tier 1
platforms, though. All cross-compiled platforms will not be receiving an
Emscripten backend yet.

This commit means that when you download the rustc package in Rustup for Tier
1 platforms you'll be receiving two trans backends, one for Emscripten and one
that's the general LLVM backend. If you never compile for Emscripten you'll
never use the Emscripten backend, so we may update this one day to only download
the Emscripten backend when you add the Emscripten target. For now though it's
just an extra 10MB gzip'd.

Closes #46819

rust-highfive · 2018-01-25T04:25:18Z

r? @pnkfelix

(rust_highfive has picked a reviewer for you, use r? to override)

alexcrichton · 2018-01-25T04:25:36Z

r? @Mark-Simulacrum

cc @rust-lang/compiler, @rust-lang/infra

kennytm · 2018-01-25T07:40:45Z

Lots of tidy error. src/llvm-emscripten needs to be excluded from tidy.

Mark-Simulacrum

So I think I'm mostly happy with this (only reviewing the second commit). Can we check if this will by-default make a clone of rust-lang/rust with ./x.py run increase by ~700 MB? I know that for me at least .git/modules/src/llvm is fairly large... I'd like to avoid that being doubled for emscripten. Not a deal breaker though.

Mark-Simulacrum · 2018-01-25T21:36:20Z

.gitmodules

+[submodule "src/llvm-emscripten"]
+	path = src/llvm-emscripten
+	url = https://github.com/rust-lang/llvm
+	branch = rust-llvm-release-4-0-1


Shouldn't this be an Emscripten branch? It seems odd that emscripten has branch = where other submodules don't.

Indeed! I figured that for now it's probably easiest to do the smallest delta possible which is to switch to the same version we have right now. I think if we upgrade in the future though we should try to depend directly on Emscripten's fork of LLVM.

As for the branch = I think that's because I passed a -b argument to git submodule add, but as far as I can tell this is never used when cloning (as the submodule has a revision). It may be used for shallow clones of submodules but I never got that working historically..

Mark-Simulacrum · 2018-01-25T21:38:32Z

src/bootstrap/compile.rs

+    const DEFAULT: bool = true;
+
+    fn should_run(run: ShouldRun) -> ShouldRun {
+        run.path("src/librustc_trans").krate("rustc_trans")


krate here is useless in theory since that crate doesn't have a crate graph really in src/ AFAIK. Though I guess it doesn't hurt... I'd prefer that we didn't have it though.

Mark-Simulacrum · 2018-01-25T21:39:16Z

src/bootstrap/compile.rs

+        run.builder.ensure(CodegenBackend {
+            compiler: run.builder.compiler(run.builder.top_stage, run.host),
+            target: run.target,
+            backend: INTERNER.intern_str("llvm"),


This should not be hardcoded here... I would expect this sort of thing in src/bootstrap/config.rs probably.

Mark-Simulacrum · 2018-01-25T21:40:39Z

src/bootstrap/compile.rs

    }
    if max > stamp_mtime {
        build.verbose(&format!("updating {:?} as {:?} changed", stamp, max_path));
    } else {
        build.verbose(&format!("updating {:?} as deps changed", stamp));
    }
    t!(t!(File::create(stamp)).write_all(&new_contents));
+    return deps


nit: just deps

Mark-Simulacrum · 2018-01-25T21:41:47Z

src/bootstrap/config.rs

@@ -318,6 +320,7 @@ impl Config {
        config.ignore_git = false;
        config.rust_dist_src = true;
        config.test_miri = false;
+        config.rust_codegen_backends = vec![INTERNER.intern_str("llvm")];


Yeah, so as I mentioned before I'd expect us to use the first element perhaps from here or something like that in the default backend to compile for in src/bootstrap/compile.rs CodegenBackend.

alexcrichton · 2018-01-25T22:59:26Z

Can we check if this will by-default make a clone of rust-lang/rust with ./x.py run increase by ~700 MB?

Indeed! I made sure to verify that we don't clone LLVM twice (we don't) and the right thing happens on CI (download from github).

Now when you do enable emscripten I think that the two llvm submodules probably won't share objects, but I'm not sure there's much we can do about that...

bors · 2018-01-26T20:35:26Z

☔ The latest upstream changes (presumably #47748) made this pull request unmergeable. Please resolve the merge conflicts.

Mark-Simulacrum · 2018-01-26T21:55:29Z

Sounds good to me. r=me; I agree that it seems unlikely that we can get git to understand that they're fundamentally the same repository. And that may not be true indefinitely.

alexcrichton · 2018-01-26T23:07:42Z

Alright! I'd like to land #47671 before this one so I'm going to hold off on rebasing this for now until #47671 lands.

alexcrichton · 2018-01-28T06:27:22Z

@bors: r=Mark-Simulacrum

bors · 2018-01-28T06:27:22Z

📌 Commit d061f30 has been approved by Mark-Simulacrum

bors · 2018-01-28T06:34:20Z

⌛ Testing commit d061f3065f705142a683a4c8b38896d14cdeedb3 with merge ebf5612396edec2dd017b015eca09cfbeadfe3dd...

bors · 2018-01-28T07:44:11Z

💔 Test failed - status-travis

alexcrichton · 2018-01-28T18:18:45Z

@bors: r=Mark-Simulacrum

bors · 2018-01-28T18:18:45Z

📌 Commit b746667 has been approved by Mark-Simulacrum

rustc: Upgrade to LLVM 6 The following submodules have been updated for a new version of LLVM: - `src/llvm` - `src/libcompiler_builtins` - transitively contains compiler-rt - `src/dlmalloc` This also updates the docker container for dist-i686-freebsd as the old 16.04 container is no longer capable of building LLVM. The compiler-rt/compiler-builtins and dlmalloc updates are pretty routine without much interesting happening, but the LLVM update here is of particular note. Unlike previous updates I haven't cherry-picked all existing patches we had on top of our LLVM branch as we have a [huge amount][patches4] and have at this point forgotten what most of them are for. Instead I started from the current `release_60` branch in LLVM and only applied patches that were necessary to get our tests working and building. The [current set of custom rustc-specific patches](rust-lang/llvm@f128612...rust-llvm-release-6-0-0) included in this LLVM update are: * rust-lang/llvm@1187443 - this is how we actually implement `cfg(target_feature)` for now and continues to not be upstreamed. While a hazard for SIMD stabilization this commit is otherwise keeping the status quo of a small rustc-specific feature. * rust-lang/llvm@013f2ec - this is a rustc-specific optimization that we haven't upstreamed, notably teaching LLVM about our allocation-related routines (which aren't malloc/free). Once we stabilize the global allocator routines we will likely want to upstream this patch, but for now it seems reasonable to keep it on our fork. * rust-lang/llvm@a65bbfd - I found this necessary to fix compilation of LLVM in our 32-bit linux container. I'm not really sure why it's necessary but my guess is that it's because of the absolutely ancient glibc that we're using. In any case it's only updating pieces we're not actually using in LLVM so I'm hoping it'll turn out alright. This doesn't seem like something we'll want to upstream.c * rust-lang/llvm@77ab1f0 - this is what's actually enabling LLVM to build in our i686-freebsd container, I'm not really sure what's going on but we for sure probably don't want to upstream this and otherwise it seems not too bad for now at least. * rust-lang/llvm@9eb9267 - we currently suffer on MSVC from an [upstream bug] which although diagnosed to a particular revision isn't currently fixed upstream (and the bug itself doesn't seem too active). This commit is a partial revert of the suspected cause of this regression (found via a bisection). I'm sort of hoping that this eventually gets fixed upstream with a similar fix (which we can replace in our branch), but for now I'm also hoping it's a relatively harmless change to have. After applying these patches (plus one [backport] which should be [backported upstream][llvm-back]) I believe we should have all tests working on all platforms in our current test suite. I'm like 99% sure that we'll need some more backports as issues are reported for LLVM 6 when this propagates through nightlies, but that's sort of just par for the course nowadays! In any case though some extra scrutiny of the patches here would definitely be welcome, along with scrutiny of the "missing patches" like a [change to pass manager order](rust-lang/llvm@2717444), [another change to pass manager order](rust-lang/llvm@c782feb), some [compile fixes for sparc](rust-lang/llvm@1a83de6), and some [fixes for solaris](rust-lang/llvm@c2bfe0a). [patches4]: rust-lang/llvm@5401fdf...rust-llvm-release-4-0-1 [backport]: rust-lang/llvm@5c54c25 [llvm-back]: https://bugs.llvm.org/show_bug.cgi?id=36114 [upstream bug]: https://bugs.llvm.org/show_bug.cgi?id=36096 --- The update to LLVM 6 is desirable for a number of reasons, notably: * This'll allow us to keep up with the upstream wasm backend, picking up new features as they start landing. * Upstream LLVM has fixed a number of SIMD-related compilation errors, especially around AVX-512 and such. * There's a few assorted known bugs which are fixed in LLVM 5 and aren't fixed in the LLVM 4 branch we're using. * Overall it's not a great idea to stagnate with our codegen backend! This update is mostly powered by #47730 which is allowing us to update LLVM *independent* of the version of LLVM that Emscripten is locked to. This means that when compiling code for Emscripten we'll still be using the old LLVM 4 backend, but when compiling code for any other target we'll be using the new LLVM 6 target. Once Emscripten updates we may no longer need this distinction, but we're not sure when that will happen! Closes #43370 Closes #43418 Closes #47015 Closes #47683 Closes rust-lang/stdarch#157 Closes rust-lang-nursery/rust-wasm#3

The following submodules have been updated for a new version of LLVM: - `src/llvm` - `src/libcompiler_builtins` - transitively contains compiler-rt - `src/dlmalloc` This also updates the docker container for dist-i686-freebsd as the old 16.04 container is no longer capable of building LLVM. The compiler-rt/compiler-builtins and dlmalloc updates are pretty routine without much interesting happening, but the LLVM update here is of particular note. Unlike previous updates I haven't cherry-picked all existing patches we had on top of our LLVM branch as we have a [huge amount][patches4] and have at this point forgotten what most of them are for. Instead I started from the current `release_60` branch in LLVM and only applied patches that were necessary to get our tests working and building. The current set of custom rustc-specific patches included in this LLVM update are: * rust-lang/llvm@1187443 - this is how we actually implement `cfg(target_feature)` for now and continues to not be upstreamed. While a hazard for SIMD stabilization this commit is otherwise keeping the status quo of a small rustc-specific feature. * rust-lang/llvm@013f2ec - this is a rustc-specific optimization that we haven't upstreamed, notably teaching LLVM about our allocation-related routines (which aren't malloc/free). Once we stabilize the global allocator routines we will likely want to upstream this patch, but for now it seems reasonable to keep it on our fork. * rust-lang/llvm@a65bbfd - I found this necessary to fix compilation of LLVM in our 32-bit linux container. I'm not really sure why it's necessary but my guess is that it's because of the absolutely ancient glibc that we're using. In any case it's only updating pieces we're not actually using in LLVM so I'm hoping it'll turn out alright. This doesn't seem like something we'll want to upstream.c * rust-lang/llvm@77ab1f0 - this is what's actually enabling LLVM to build in our i686-freebsd container, I'm not really sure what's going on but we for sure probably don't want to upstream this and otherwise it seems not too bad for now at least. * rust-lang/llvm@9eb9267 - we currently suffer on MSVC from an [upstream bug] which although diagnosed to a particular revision isn't currently fixed upstream (and the bug itself doesn't seem too active). This commit is a partial revert of the suspected cause of this regression (found via a bisection). I'm sort of hoping that this eventually gets fixed upstream with a similar fix (which we can replace in our branch), but for now I'm also hoping it's a relatively harmless change to have. After applying these patches (plus one [backport] which should be [backported upstream][llvm-back]) I believe we should have all tests working on all platforms in our current test suite. I'm like 99% sure that we'll need some more backports as issues are reported for LLVM 6 when this propagates through nightlies, but that's sort of just par for the course nowadays! In any case though some extra scrutiny of the patches here would definitely be welcome, along with scrutiny of the "missing patches" like a [change to pass manager order](rust-lang/llvm@27174447533), [another change to pass manager order](rust-lang/llvm@c782febb7b9), some [compile fixes for sparc](rust-lang/llvm@1a83de63c42), and some [fixes for solaris](rust-lang/llvm@c2bfe0abb). [patches4]: rust-lang/llvm@5401fdf...rust-llvm-release-4-0-1 [backport]: rust-lang/llvm@5c54c252db [llvm-back]: https://bugs.llvm.org/show_bug.cgi?id=36114 [upstream bug]: https://bugs.llvm.org/show_bug.cgi?id=36096 --- The update to LLVM 6 is desirable for a number of reasons, notably: * This'll allow us to keep up with the upstream wasm backend, picking up new features as they start landing. * Upstream LLVM has fixed a number of SIMD-related compilation errors, especially around AVX-512 and such. * There's a few assorted known bugs which are fixed in LLVM 5 and aren't fixed in the LLVM 4 branch we're using. * Overall it's not a great idea to stagnate with our codegen backend! This update is mostly powered by rust-lang#47730 which is allowing us to update LLVM *independent* of the version of LLVM that Emscripten is locked to. This means that when compiling code for Emscripten we'll still be using the old LLVM 4 backend, but when compiling code for any other target we'll be using the new LLVM 6 target. Once Emscripten updates we may no longer need this distinction, but we're not sure when that will happen! Closes rust-lang#43370 Closes rust-lang#43418 Closes rust-lang#47015 Closes rust-lang#47683 Closes rust-lang/stdarch#157 Closes rust-lang-nursery/rust-wasm#3

rustc: Upgrade to LLVM 6 The following submodules have been updated for a new version of LLVM: - `src/llvm` - `src/libcompiler_builtins` - transitively contains compiler-rt - `src/dlmalloc` This also updates the docker container for dist-i686-freebsd as the old 16.04 container is no longer capable of building LLVM. The compiler-rt/compiler-builtins and dlmalloc updates are pretty routine without much interesting happening, but the LLVM update here is of particular note. Unlike previous updates I haven't cherry-picked all existing patches we had on top of our LLVM branch as we have a [huge amount][patches4] and have at this point forgotten what most of them are for. Instead I started from the current `release_60` branch in LLVM and only applied patches that were necessary to get our tests working and building. The [current set of custom rustc-specific patches](rust-lang/llvm@f128612...rust-llvm-release-6-0-0) included in this LLVM update are: * rust-lang/llvm@1187443 - this is how we actually implement `cfg(target_feature)` for now and continues to not be upstreamed. While a hazard for SIMD stabilization this commit is otherwise keeping the status quo of a small rustc-specific feature. * rust-lang/llvm@013f2ec - this is a rustc-specific optimization that we haven't upstreamed, notably teaching LLVM about our allocation-related routines (which aren't malloc/free). Once we stabilize the global allocator routines we will likely want to upstream this patch, but for now it seems reasonable to keep it on our fork. * rust-lang/llvm@a65bbfd - I found this necessary to fix compilation of LLVM in our 32-bit linux container. I'm not really sure why it's necessary but my guess is that it's because of the absolutely ancient glibc that we're using. In any case it's only updating pieces we're not actually using in LLVM so I'm hoping it'll turn out alright. This doesn't seem like something we'll want to upstream.c * rust-lang/llvm@77ab1f0 - this is what's actually enabling LLVM to build in our i686-freebsd container, I'm not really sure what's going on but we for sure probably don't want to upstream this and otherwise it seems not too bad for now at least. * rust-lang/llvm@9eb9267 - we currently suffer on MSVC from an [upstream bug] which although diagnosed to a particular revision isn't currently fixed upstream (and the bug itself doesn't seem too active). This commit is a partial revert of the suspected cause of this regression (found via a bisection). I'm sort of hoping that this eventually gets fixed upstream with a similar fix (which we can replace in our branch), but for now I'm also hoping it's a relatively harmless change to have. After applying these patches (plus one [backport] which should be [backported upstream][llvm-back]) I believe we should have all tests working on all platforms in our current test suite. I'm like 99% sure that we'll need some more backports as issues are reported for LLVM 6 when this propagates through nightlies, but that's sort of just par for the course nowadays! In any case though some extra scrutiny of the patches here would definitely be welcome, along with scrutiny of the "missing patches" like a [change to pass manager order](rust-lang/llvm@2717444), [another change to pass manager order](rust-lang/llvm@c782feb), some [compile fixes for sparc](rust-lang/llvm@1a83de6), and some [fixes for solaris](rust-lang/llvm@c2bfe0a). [patches4]: rust-lang/llvm@5401fdf...rust-llvm-release-4-0-1 [backport]: rust-lang/llvm@5c54c25 [llvm-back]: https://bugs.llvm.org/show_bug.cgi?id=36114 [upstream bug]: https://bugs.llvm.org/show_bug.cgi?id=36096 --- The update to LLVM 6 is desirable for a number of reasons, notably: * This'll allow us to keep up with the upstream wasm backend, picking up new features as they start landing. * Upstream LLVM has fixed a number of SIMD-related compilation errors, especially around AVX-512 and such. * There's a few assorted known bugs which are fixed in LLVM 5 and aren't fixed in the LLVM 4 branch we're using. * Overall it's not a great idea to stagnate with our codegen backend! This update is mostly powered by #47730 which is allowing us to update LLVM *independent* of the version of LLVM that Emscripten is locked to. This means that when compiling code for Emscripten we'll still be using the old LLVM 4 backend, but when compiling code for any other target we'll be using the new LLVM 6 target. Once Emscripten updates we may no longer need this distinction, but we're not sure when that will happen! Closes #43370 Closes #43418 Closes #47015 Closes #47683 Closes rust-lang/stdarch#157 Closes rust-lang-nursery/rust-wasm#3

The following submodules have been updated for a new version of LLVM: - `src/llvm` - `src/libcompiler_builtins` - transitively contains compiler-rt - `src/dlmalloc` This also updates the docker container for dist-i686-freebsd as the old 16.04 container is no longer capable of building LLVM. The compiler-rt/compiler-builtins and dlmalloc updates are pretty routine without much interesting happening, but the LLVM update here is of particular note. Unlike previous updates I haven't cherry-picked all existing patches we had on top of our LLVM branch as we have a [huge amount][patches4] and have at this point forgotten what most of them are for. Instead I started from the current `release_60` branch in LLVM and only applied patches that were necessary to get our tests working and building. The current set of custom rustc-specific patches included in this LLVM update are: * rust-lang/llvm@1187443 - this is how we actually implement `cfg(target_feature)` for now and continues to not be upstreamed. While a hazard for SIMD stabilization this commit is otherwise keeping the status quo of a small rustc-specific feature. * rust-lang/llvm@013f2ec - this is a rustc-specific optimization that we haven't upstreamed, notably teaching LLVM about our allocation-related routines (which aren't malloc/free). Once we stabilize the global allocator routines we will likely want to upstream this patch, but for now it seems reasonable to keep it on our fork. * rust-lang/llvm@a65bbfd - I found this necessary to fix compilation of LLVM in our 32-bit linux container. I'm not really sure why it's necessary but my guess is that it's because of the absolutely ancient glibc that we're using. In any case it's only updating pieces we're not actually using in LLVM so I'm hoping it'll turn out alright. This doesn't seem like something we'll want to upstream.c * rust-lang/llvm@77ab1f0 - this is what's actually enabling LLVM to build in our i686-freebsd container, I'm not really sure what's going on but we for sure probably don't want to upstream this and otherwise it seems not too bad for now at least. * rust-lang/llvm@9eb9267 - we currently suffer on MSVC from an [upstream bug] which although diagnosed to a particular revision isn't currently fixed upstream (and the bug itself doesn't seem too active). This commit is a partial revert of the suspected cause of this regression (found via a bisection). I'm sort of hoping that this eventually gets fixed upstream with a similar fix (which we can replace in our branch), but for now I'm also hoping it's a relatively harmless change to have. After applying these patches (plus one [backport] which should be [backported upstream][llvm-back]) I believe we should have all tests working on all platforms in our current test suite. I'm like 99% sure that we'll need some more backports as issues are reported for LLVM 6 when this propagates through nightlies, but that's sort of just par for the course nowadays! In any case though some extra scrutiny of the patches here would definitely be welcome, along with scrutiny of the "missing patches" like a [change to pass manager order](rust-lang/llvm@27174447533), [another change to pass manager order](rust-lang/llvm@c782febb7b9), some [compile fixes for sparc](rust-lang/llvm@1a83de63c42), and some [fixes for solaris](rust-lang/llvm@c2bfe0abb). [patches4]: rust-lang/llvm@5401fdf...rust-llvm-release-4-0-1 [backport]: rust-lang/llvm@5c54c252db [llvm-back]: https://bugs.llvm.org/show_bug.cgi?id=36114 [upstream bug]: https://bugs.llvm.org/show_bug.cgi?id=36096 --- The update to LLVM 6 is desirable for a number of reasons, notably: * This'll allow us to keep up with the upstream wasm backend, picking up new features as they start landing. * Upstream LLVM has fixed a number of SIMD-related compilation errors, especially around AVX-512 and such. * There's a few assorted known bugs which are fixed in LLVM 5 and aren't fixed in the LLVM 4 branch we're using. * Overall it's not a great idea to stagnate with our codegen backend! This update is mostly powered by #47730 which is allowing us to update LLVM *independent* of the version of LLVM that Emscripten is locked to. This means that when compiling code for Emscripten we'll still be using the old LLVM 4 backend, but when compiling code for any other target we'll be using the new LLVM 6 target. Once Emscripten updates we may no longer need this distinction, but we're not sure when that will happen! Closes #43370 Closes #43418 Closes #47015 Closes #47683 Closes rust-lang/stdarch#157 Closes rust-lang-nursery/rust-wasm#3

The following submodules have been updated for a new version of LLVM: - `src/llvm` - `src/libcompiler_builtins` - transitively contains compiler-rt - `src/dlmalloc` This also updates the docker container for dist-i686-freebsd as the old 16.04 container is no longer capable of building LLVM. The compiler-rt/compiler-builtins and dlmalloc updates are pretty routine without much interesting happening, but the LLVM update here is of particular note. Unlike previous updates I haven't cherry-picked all existing patches we had on top of our LLVM branch as we have a [huge amount][patches4] and have at this point forgotten what most of them are for. Instead I started from the current `release_60` branch in LLVM and only applied patches that were necessary to get our tests working and building. The current set of custom rustc-specific patches included in this LLVM update are: * rust-lang/llvm@1187443 - this is how we actually implement `cfg(target_feature)` for now and continues to not be upstreamed. While a hazard for SIMD stabilization this commit is otherwise keeping the status quo of a small rustc-specific feature. * rust-lang/llvm@013f2ec - this is a rustc-specific optimization that we haven't upstreamed, notably teaching LLVM about our allocation-related routines (which aren't malloc/free). Once we stabilize the global allocator routines we will likely want to upstream this patch, but for now it seems reasonable to keep it on our fork. * rust-lang/llvm@a65bbfd - I found this necessary to fix compilation of LLVM in our 32-bit linux container. I'm not really sure why it's necessary but my guess is that it's because of the absolutely ancient glibc that we're using. In any case it's only updating pieces we're not actually using in LLVM so I'm hoping it'll turn out alright. This doesn't seem like something we'll want to upstream.c * rust-lang/llvm@77ab1f0 - this is what's actually enabling LLVM to build in our i686-freebsd container, I'm not really sure what's going on but we for sure probably don't want to upstream this and otherwise it seems not too bad for now at least. * rust-lang/llvm@9eb9267 - we currently suffer on MSVC from an [upstream bug] which although diagnosed to a particular revision isn't currently fixed upstream (and the bug itself doesn't seem too active). This commit is a partial revert of the suspected cause of this regression (found via a bisection). I'm sort of hoping that this eventually gets fixed upstream with a similar fix (which we can replace in our branch), but for now I'm also hoping it's a relatively harmless change to have. After applying these patches (plus one [backport] which should be [backported upstream][llvm-back]) I believe we should have all tests working on all platforms in our current test suite. I'm like 99% sure that we'll need some more backports as issues are reported for LLVM 6 when this propagates through nightlies, but that's sort of just par for the course nowadays! In any case though some extra scrutiny of the patches here would definitely be welcome, along with scrutiny of the "missing patches" like a [change to pass manager order](rust-lang/llvm@27174447533), [another change to pass manager order](rust-lang/llvm@c782febb7b9), some [compile fixes for sparc](rust-lang/llvm@1a83de63c42), and some [fixes for solaris](rust-lang/llvm@c2bfe0abb). [patches4]: rust-lang/llvm@5401fdf...rust-llvm-release-4-0-1 [backport]: rust-lang/llvm@5c54c252db [llvm-back]: https://bugs.llvm.org/show_bug.cgi?id=36114 [upstream bug]: https://bugs.llvm.org/show_bug.cgi?id=36096 --- The update to LLVM 6 is desirable for a number of reasons, notably: * This'll allow us to keep up with the upstream wasm backend, picking up new features as they start landing. * Upstream LLVM has fixed a number of SIMD-related compilation errors, especially around AVX-512 and such. * There's a few assorted known bugs which are fixed in LLVM 5 and aren't fixed in the LLVM 4 branch we're using. * Overall it's not a great idea to stagnate with our codegen backend! This update is mostly powered by rust-lang#47730 which is allowing us to update LLVM *independent* of the version of LLVM that Emscripten is locked to. This means that when compiling code for Emscripten we'll still be using the old LLVM 4 backend, but when compiling code for any other target we'll be using the new LLVM 6 target. Once Emscripten updates we may no longer need this distinction, but we're not sure when that will happen! Closes rust-lang#43370 Closes rust-lang#43418 Closes rust-lang#47015 Closes rust-lang#47683 Closes rust-lang/stdarch#157 Closes rust-lang-nursery/rust-wasm#3

rustc: Upgrade to LLVM 6 The following submodules have been updated for a new version of LLVM: - `src/llvm` - `src/libcompiler_builtins` - transitively contains compiler-rt - `src/dlmalloc` This also updates the docker container for dist-i686-freebsd as the old 16.04 container is no longer capable of building LLVM. The compiler-rt/compiler-builtins and dlmalloc updates are pretty routine without much interesting happening, but the LLVM update here is of particular note. Unlike previous updates I haven't cherry-picked all existing patches we had on top of our LLVM branch as we have a [huge amount][patches4] and have at this point forgotten what most of them are for. Instead I started from the current `release_60` branch in LLVM and only applied patches that were necessary to get our tests working and building. The [current set of custom rustc-specific patches](rust-lang/llvm@f128612...rust-llvm-release-6-0-0) included in this LLVM update are: * rust-lang/llvm@1187443 - this is how we actually implement `cfg(target_feature)` for now and continues to not be upstreamed. While a hazard for SIMD stabilization this commit is otherwise keeping the status quo of a small rustc-specific feature. * rust-lang/llvm@013f2ec - this is a rustc-specific optimization that we haven't upstreamed, notably teaching LLVM about our allocation-related routines (which aren't malloc/free). Once we stabilize the global allocator routines we will likely want to upstream this patch, but for now it seems reasonable to keep it on our fork. * rust-lang/llvm@a65bbfd - I found this necessary to fix compilation of LLVM in our 32-bit linux container. I'm not really sure why it's necessary but my guess is that it's because of the absolutely ancient glibc that we're using. In any case it's only updating pieces we're not actually using in LLVM so I'm hoping it'll turn out alright. This doesn't seem like something we'll want to upstream.c * rust-lang/llvm@77ab1f0 - this is what's actually enabling LLVM to build in our i686-freebsd container, I'm not really sure what's going on but we for sure probably don't want to upstream this and otherwise it seems not too bad for now at least. * rust-lang/llvm@9eb9267 - we currently suffer on MSVC from an [upstream bug] which although diagnosed to a particular revision isn't currently fixed upstream (and the bug itself doesn't seem too active). This commit is a partial revert of the suspected cause of this regression (found via a bisection). I'm sort of hoping that this eventually gets fixed upstream with a similar fix (which we can replace in our branch), but for now I'm also hoping it's a relatively harmless change to have. After applying these patches (plus one [backport] which should be [backported upstream][llvm-back]) I believe we should have all tests working on all platforms in our current test suite. I'm like 99% sure that we'll need some more backports as issues are reported for LLVM 6 when this propagates through nightlies, but that's sort of just par for the course nowadays! In any case though some extra scrutiny of the patches here would definitely be welcome, along with scrutiny of the "missing patches" like a [change to pass manager order](rust-lang/llvm@2717444), [another change to pass manager order](rust-lang/llvm@c782feb), some [compile fixes for sparc](rust-lang/llvm@1a83de6), and some [fixes for solaris](rust-lang/llvm@c2bfe0a). [patches4]: rust-lang/llvm@5401fdf...rust-llvm-release-4-0-1 [backport]: rust-lang/llvm@5c54c25 [llvm-back]: https://bugs.llvm.org/show_bug.cgi?id=36114 [upstream bug]: https://bugs.llvm.org/show_bug.cgi?id=36096 --- The update to LLVM 6 is desirable for a number of reasons, notably: * This'll allow us to keep up with the upstream wasm backend, picking up new features as they start landing. * Upstream LLVM has fixed a number of SIMD-related compilation errors, especially around AVX-512 and such. * There's a few assorted known bugs which are fixed in LLVM 5 and aren't fixed in the LLVM 4 branch we're using. * Overall it's not a great idea to stagnate with our codegen backend! This update is mostly powered by #47730 which is allowing us to update LLVM *independent* of the version of LLVM that Emscripten is locked to. This means that when compiling code for Emscripten we'll still be using the old LLVM 4 backend, but when compiling code for any other target we'll be using the new LLVM 6 target. Once Emscripten updates we may no longer need this distinction, but we're not sure when that will happen! Closes #43370 Closes #43418 Closes #47015 Closes #47683 Closes rust-lang/stdarch#157 Closes rust-lang-nursery/rust-wasm#3

The following submodules have been updated for a new version of LLVM: - `src/llvm` - `src/libcompiler_builtins` - transitively contains compiler-rt - `src/dlmalloc` This also updates the docker container for dist-i686-freebsd as the old 16.04 container is no longer capable of building LLVM. The compiler-rt/compiler-builtins and dlmalloc updates are pretty routine without much interesting happening, but the LLVM update here is of particular note. Unlike previous updates I haven't cherry-picked all existing patches we had on top of our LLVM branch as we have a [huge amount][patches4] and have at this point forgotten what most of them are for. Instead I started from the current `release_60` branch in LLVM and only applied patches that were necessary to get our tests working and building. The current set of custom rustc-specific patches included in this LLVM update are: * rust-lang/llvm@1187443 - this is how we actually implement `cfg(target_feature)` for now and continues to not be upstreamed. While a hazard for SIMD stabilization this commit is otherwise keeping the status quo of a small rustc-specific feature. * rust-lang/llvm@013f2ec - this is a rustc-specific optimization that we haven't upstreamed, notably teaching LLVM about our allocation-related routines (which aren't malloc/free). Once we stabilize the global allocator routines we will likely want to upstream this patch, but for now it seems reasonable to keep it on our fork. * rust-lang/llvm@a65bbfd - I found this necessary to fix compilation of LLVM in our 32-bit linux container. I'm not really sure why it's necessary but my guess is that it's because of the absolutely ancient glibc that we're using. In any case it's only updating pieces we're not actually using in LLVM so I'm hoping it'll turn out alright. This doesn't seem like something we'll want to upstream.c * rust-lang/llvm@77ab1f0 - this is what's actually enabling LLVM to build in our i686-freebsd container, I'm not really sure what's going on but we for sure probably don't want to upstream this and otherwise it seems not too bad for now at least. * rust-lang/llvm@9eb9267 - we currently suffer on MSVC from an [upstream bug] which although diagnosed to a particular revision isn't currently fixed upstream (and the bug itself doesn't seem too active). This commit is a partial revert of the suspected cause of this regression (found via a bisection). I'm sort of hoping that this eventually gets fixed upstream with a similar fix (which we can replace in our branch), but for now I'm also hoping it's a relatively harmless change to have. After applying these patches (plus one [backport] which should be [backported upstream][llvm-back]) I believe we should have all tests working on all platforms in our current test suite. I'm like 99% sure that we'll need some more backports as issues are reported for LLVM 6 when this propagates through nightlies, but that's sort of just par for the course nowadays! In any case though some extra scrutiny of the patches here would definitely be welcome, along with scrutiny of the "missing patches" like a [change to pass manager order](rust-lang/llvm@27174447533), [another change to pass manager order](rust-lang/llvm@c782febb7b9), some [compile fixes for sparc](rust-lang/llvm@1a83de63c42), and some [fixes for solaris](rust-lang/llvm@c2bfe0abb). [patches4]: rust-lang/llvm@5401fdf...rust-llvm-release-4-0-1 [backport]: rust-lang/llvm@5c54c252db [llvm-back]: https://bugs.llvm.org/show_bug.cgi?id=36114 [upstream bug]: https://bugs.llvm.org/show_bug.cgi?id=36096 --- The update to LLVM 6 is desirable for a number of reasons, notably: * This'll allow us to keep up with the upstream wasm backend, picking up new features as they start landing. * Upstream LLVM has fixed a number of SIMD-related compilation errors, especially around AVX-512 and such. * There's a few assorted known bugs which are fixed in LLVM 5 and aren't fixed in the LLVM 4 branch we're using. * Overall it's not a great idea to stagnate with our codegen backend! This update is mostly powered by rust-lang#47730 which is allowing us to update LLVM *independent* of the version of LLVM that Emscripten is locked to. This means that when compiling code for Emscripten we'll still be using the old LLVM 4 backend, but when compiling code for any other target we'll be using the new LLVM 6 target. Once Emscripten updates we may no longer need this distinction, but we're not sure when that will happen! Closes rust-lang#43370 Closes rust-lang#43418 Closes rust-lang#47015 Closes rust-lang#47683 Closes rust-lang/stdarch#157 Closes rust-lang-nursery/rust-wasm#3

rustc: Upgrade to LLVM 6 The following submodules have been updated for a new version of LLVM: - `src/llvm` - `src/libcompiler_builtins` - transitively contains compiler-rt - `src/dlmalloc` This also updates the docker container for dist-i686-freebsd as the old 16.04 container is no longer capable of building LLVM. The compiler-rt/compiler-builtins and dlmalloc updates are pretty routine without much interesting happening, but the LLVM update here is of particular note. Unlike previous updates I haven't cherry-picked all existing patches we had on top of our LLVM branch as we have a [huge amount][patches4] and have at this point forgotten what most of them are for. Instead I started from the current `release_60` branch in LLVM and only applied patches that were necessary to get our tests working and building. The [current set of custom rustc-specific patches](rust-lang/llvm@f128612...rust-llvm-release-6-0-0) included in this LLVM update are: * rust-lang/llvm@1187443 - this is how we actually implement `cfg(target_feature)` for now and continues to not be upstreamed. While a hazard for SIMD stabilization this commit is otherwise keeping the status quo of a small rustc-specific feature. * rust-lang/llvm@013f2ec - this is a rustc-specific optimization that we haven't upstreamed, notably teaching LLVM about our allocation-related routines (which aren't malloc/free). Once we stabilize the global allocator routines we will likely want to upstream this patch, but for now it seems reasonable to keep it on our fork. * rust-lang/llvm@a65bbfd - I found this necessary to fix compilation of LLVM in our 32-bit linux container. I'm not really sure why it's necessary but my guess is that it's because of the absolutely ancient glibc that we're using. In any case it's only updating pieces we're not actually using in LLVM so I'm hoping it'll turn out alright. This doesn't seem like something we'll want to upstream.c * rust-lang/llvm@77ab1f0 - this is what's actually enabling LLVM to build in our i686-freebsd container, I'm not really sure what's going on but we for sure probably don't want to upstream this and otherwise it seems not too bad for now at least. * rust-lang/llvm@9eb9267 - we currently suffer on MSVC from an [upstream bug] which although diagnosed to a particular revision isn't currently fixed upstream (and the bug itself doesn't seem too active). This commit is a partial revert of the suspected cause of this regression (found via a bisection). I'm sort of hoping that this eventually gets fixed upstream with a similar fix (which we can replace in our branch), but for now I'm also hoping it's a relatively harmless change to have. After applying these patches (plus one [backport] which should be [backported upstream][llvm-back]) I believe we should have all tests working on all platforms in our current test suite. I'm like 99% sure that we'll need some more backports as issues are reported for LLVM 6 when this propagates through nightlies, but that's sort of just par for the course nowadays! In any case though some extra scrutiny of the patches here would definitely be welcome, along with scrutiny of the "missing patches" like a [change to pass manager order](rust-lang/llvm@2717444), [another change to pass manager order](rust-lang/llvm@c782feb), some [compile fixes for sparc](rust-lang/llvm@1a83de6), and some [fixes for solaris](rust-lang/llvm@c2bfe0a). [patches4]: rust-lang/llvm@5401fdf...rust-llvm-release-4-0-1 [backport]: rust-lang/llvm@5c54c25 [llvm-back]: https://bugs.llvm.org/show_bug.cgi?id=36114 [upstream bug]: https://bugs.llvm.org/show_bug.cgi?id=36096 --- The update to LLVM 6 is desirable for a number of reasons, notably: * This'll allow us to keep up with the upstream wasm backend, picking up new features as they start landing. * Upstream LLVM has fixed a number of SIMD-related compilation errors, especially around AVX-512 and such. * There's a few assorted known bugs which are fixed in LLVM 5 and aren't fixed in the LLVM 4 branch we're using. * Overall it's not a great idea to stagnate with our codegen backend! This update is mostly powered by #47730 which is allowing us to update LLVM *independent* of the version of LLVM that Emscripten is locked to. This means that when compiling code for Emscripten we'll still be using the old LLVM 4 backend, but when compiling code for any other target we'll be using the new LLVM 6 target. Once Emscripten updates we may no longer need this distinction, but we're not sure when that will happen! Closes #43370 Closes #43418 Closes #47015 Closes #47683 Closes rust-lang/stdarch#157 Closes rust-lang-nursery/rust-wasm#3

rust-highfive assigned pnkfelix Jan 25, 2018

rust-highfive assigned Mark-Simulacrum and unassigned pnkfelix Jan 25, 2018

alexcrichton mentioned this pull request Jan 25, 2018

Implement shipping a per-target LLVM backend #46819

Closed

alexcrichton force-pushed the multitrans branch from 885a792 to 2ab5374 Compare January 25, 2018 16:29

Mark-Simulacrum reviewed Jan 25, 2018

View reviewed changes

alexcrichton force-pushed the multitrans branch from 2ab5374 to 0ef3ab7 Compare January 25, 2018 22:58

alexcrichton added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 26, 2018

alexcrichton force-pushed the multitrans branch from 0ef3ab7 to d061f30 Compare January 28, 2018 06:26

kennytm added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 28, 2018

alexcrichton force-pushed the multitrans branch from d061f30 to b746667 Compare January 28, 2018 18:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustc: Split Emscripten to a separate codegen backend #47730

rustc: Split Emscripten to a separate codegen backend #47730

alexcrichton commented Jan 25, 2018

rust-highfive commented Jan 25, 2018

alexcrichton commented Jan 25, 2018

kennytm commented Jan 25, 2018

Mark-Simulacrum left a comment

Mark-Simulacrum Jan 25, 2018

alexcrichton Jan 25, 2018

Mark-Simulacrum Jan 25, 2018

Mark-Simulacrum Jan 25, 2018

Mark-Simulacrum Jan 25, 2018

Mark-Simulacrum Jan 25, 2018

alexcrichton commented Jan 25, 2018

bors commented Jan 26, 2018

Mark-Simulacrum commented Jan 26, 2018

alexcrichton commented Jan 26, 2018

alexcrichton commented Jan 28, 2018

bors commented Jan 28, 2018

bors commented Jan 28, 2018

bors commented Jan 28, 2018

alexcrichton commented Jan 28, 2018

bors commented Jan 28, 2018

rustc: Split Emscripten to a separate codegen backend #47730

rustc: Split Emscripten to a separate codegen backend #47730

Conversation

alexcrichton commented Jan 25, 2018

rust-highfive commented Jan 25, 2018

alexcrichton commented Jan 25, 2018

kennytm commented Jan 25, 2018

Mark-Simulacrum left a comment

Choose a reason for hiding this comment

Mark-Simulacrum Jan 25, 2018

Choose a reason for hiding this comment

alexcrichton Jan 25, 2018

Choose a reason for hiding this comment

Mark-Simulacrum Jan 25, 2018

Choose a reason for hiding this comment

Mark-Simulacrum Jan 25, 2018

Choose a reason for hiding this comment

Mark-Simulacrum Jan 25, 2018

Choose a reason for hiding this comment

Mark-Simulacrum Jan 25, 2018

Choose a reason for hiding this comment

alexcrichton commented Jan 25, 2018

bors commented Jan 26, 2018

Mark-Simulacrum commented Jan 26, 2018

alexcrichton commented Jan 26, 2018

alexcrichton commented Jan 28, 2018

bors commented Jan 28, 2018

bors commented Jan 28, 2018

bors commented Jan 28, 2018

alexcrichton commented Jan 28, 2018

bors commented Jan 28, 2018