Allow downloading LLVM on Windows and MacOS #80932

jyn514 · 2021-01-12T01:34:05Z

Don't ignore packaging llvm/lib/ for rust-dev when LLVM is linked
statically
Add link-type.txt so bootstrap knows whether llvm was linked
statically or dynamically
Don't assume CI LLVM is linked dynamically in bootstrap::config
Fall back to dynamic linking if link-type.txt doesn't exist
Fix existing bug that split the output of llvm-config on lines, not spaces
Only special case MacOS when dynamic linking. Static linking works fine.

Enable building LLVM tests

This works around the following llvm bug:

llvm-config: error: component libraries and shared library

llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libgtest.a
llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libgtest_main.a
llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libLLVMTestingSupport.a
thread 'main' panicked at 'command did not execute successfully: "/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/bin/llvm-config" "--libfiles"

I'm not sure why llvm-config thinks these are required, but to avoid
the error, this builds them anyway.

Bump version of download-ci-llvm-stamp

src/llvm-project hasn't changed, but the generated tarball has.

Fixes #77084.

Current Status

This works on both MacOS and Windows! 🎉 🎉 Thanks to @nagisa, @Halkcyon, @Lokathor, @jryans, and @poliorcetics for helping me test!

The if-available check now supports all tier 1 platforms. Although only x64 apple and x64 msvc have been tested, none of the changes here are Windows or Mac specific, and I expect this to work anywhere that LLVM artifacts are uploaded to CI (i.e. the rust-dev component exists).

Windows

Note that if you have an old version of MSVC build tools you'll need to update them. VS Build Tools 2019 14.28 and later are known to work. With old tools, you may see an error like the following:

error LNK2001: unresolved external symbol __imp___std_init_once_complete

rust-highfive · 2021-01-12T01:34:09Z

r? @Mark-Simulacrum

(rust-highfive has picked a reviewer for you, use r? to override)

nagisa · 2021-01-13T21:29:59Z

src/bootstrap/bootstrap.py

+        supported_platforms = [
+            "x86_64-unknown-linux-gnu",
+            # FIXME: maybe windows-gnu should be supported too?
+            "x86_64-pc-windows-msvc",


I'd be fairly comfortable putting all T1 targets on this list at this point. x86_64 working seems good enough indication that 32-bit equivalents and the aarch64 target will work too, since the complexity is in primarily in the "os" part rather than the "arch" part of the triple.

Ok, I added them to the list. Is there a way to see a list of all the artifacts that are uploaded? e.g. https://ci-artifacts.rust-lang.org/rustc-builds/695f18ebab0ca895841514da841eb3bd6f6bbd88 gives a 404.

src/bootstrap/bootstrap.py

src/bootstrap/native.rs

src/bootstrap/config.rs

src/bootstrap/dist.rs

src/bootstrap/native.rs

Mark-Simulacrum · 2021-01-16T02:17:16Z

r=me with two small comments addressed

- Don't ignore packaging `llvm/lib/` for `rust-dev` when LLVM is linked statically - Add `link-type.txt` so bootstrap knows whether llvm was linked statically or dynamically - Don't assume CI LLVM is linked dynamically in `bootstrap::config` - Fall back to dynamic linking if `link-type.txt` doesn't exist - Fix existing bug that split the output of `llvm-config` on lines, not spaces - Enable building LLVM tests This works around the following llvm bug: ``` llvm-config: error: component libraries and shared library llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libgtest.a llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libgtest_main.a llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libLLVMTestingSupport.a thread 'main' panicked at 'command did not execute successfully: "/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/bin/llvm-config" "--libfiles" ``` I'm not sure why llvm-config thinks these are required, but to avoid the error, this builds them anyway. - Temporarily set windows as the try builder. This should be reverted before merging. - Bump version of `download-ci-llvm-stamp` `src/llvm-project` hasn't changed, but the generated tarball has. - Only special case MacOS when dynamic linking. Static linking works fine. - Store `link-type.txt` to the top-level of the tarball This allows writing the link type unconditionally. Previously, bootstrap had to keep track of whether the file IO *would* succeed (it would fail if `lib/` didn't exist), which was prone to bugs. - Make `link-type.txt` required Anyone downloading this from CI should be using a version of bootstrap that matches the version of the uploaded artifacts. So a missing link-type indicates a bug in x.py.

... and update the comment in `config.toml.example`

jyn514 · 2021-01-16T03:12:12Z

@bors r=Mark-Simulacrum

Thanks for the review!

bors · 2021-01-16T03:12:14Z

📌 Commit 5c4adbe has been approved by Mark-Simulacrum

…k-Simulacrum Allow downloading LLVM on Windows and MacOS - Don't ignore packaging `llvm/lib/` for `rust-dev` when LLVM is linked statically - Add `link-type.txt` so bootstrap knows whether llvm was linked statically or dynamically - Don't assume CI LLVM is linked dynamically in `bootstrap::config` - Fall back to dynamic linking if `link-type.txt` doesn't exist - Fix existing bug that split the output of `llvm-config` on lines, not spaces - Only special case MacOS when dynamic linking. Static linking works fine. - Enable building LLVM tests This works around the following llvm bug: ``` llvm-config: error: component libraries and shared library llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libgtest.a llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libgtest_main.a llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libLLVMTestingSupport.a thread 'main' panicked at 'command did not execute successfully: "/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/bin/llvm-config" "--libfiles" ``` I'm not sure why llvm-config thinks these are required, but to avoid the error, this builds them anyway. - Bump version of `download-ci-llvm-stamp` `src/llvm-project` hasn't changed, but the generated tarball has. Fixes rust-lang#77084. # Current Status This works on both MacOS and Windows! 🎉 🎉 Thanks to `@nagisa,` `@halkcyon,` `@Lokathor,` `@jryans,` and `@poliorcetics` for helping me test! The `if-available` check now supports all tier 1 platforms. Although only x64 apple and x64 msvc have been tested, none of the changes here are Windows or Mac specific, and I expect this to work anywhere that LLVM artifacts are uploaded to CI (i.e. the `rust-dev` component exists). ## Windows Note that if you have an old version of MSVC build tools you'll need to update them. VS Build Tools 2019 14.28 and later are known to work. With old tools, you may see an error like the following: ``` error LNK2001: unresolved external symbol __imp___std_init_once_complete ```

…k-Simulacrum Allow downloading LLVM on Windows and MacOS - Don't ignore packaging `llvm/lib/` for `rust-dev` when LLVM is linked statically - Add `link-type.txt` so bootstrap knows whether llvm was linked statically or dynamically - Don't assume CI LLVM is linked dynamically in `bootstrap::config` - Fall back to dynamic linking if `link-type.txt` doesn't exist - Fix existing bug that split the output of `llvm-config` on lines, not spaces - Only special case MacOS when dynamic linking. Static linking works fine. - Enable building LLVM tests This works around the following llvm bug: ``` llvm-config: error: component libraries and shared library llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libgtest.a llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libgtest_main.a llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libLLVMTestingSupport.a thread 'main' panicked at 'command did not execute successfully: "/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/bin/llvm-config" "--libfiles" ``` I'm not sure why llvm-config thinks these are required, but to avoid the error, this builds them anyway. - Bump version of `download-ci-llvm-stamp` `src/llvm-project` hasn't changed, but the generated tarball has. Fixes rust-lang#77084. # Current Status This works on both MacOS and Windows! 🎉 🎉 Thanks to ``@nagisa,`` ``@halkcyon,`` ``@Lokathor,`` ``@jryans,`` and ``@poliorcetics`` for helping me test! The `if-available` check now supports all tier 1 platforms. Although only x64 apple and x64 msvc have been tested, none of the changes here are Windows or Mac specific, and I expect this to work anywhere that LLVM artifacts are uploaded to CI (i.e. the `rust-dev` component exists). ## Windows Note that if you have an old version of MSVC build tools you'll need to update them. VS Build Tools 2019 14.28 and later are known to work. With old tools, you may see an error like the following: ``` error LNK2001: unresolved external symbol __imp___std_init_once_complete ```

Rollup of 13 pull requests Successful merges: - rust-lang#79298 (correctly deal with late-bound lifetimes in anon consts) - rust-lang#80031 (resolve: Reject ambiguity built-in attr vs different built-in attr) - rust-lang#80201 (Add benchmark and fast path for BufReader::read_exact) - rust-lang#80635 (Improve diagnostics when closure doesn't meet trait bound) - rust-lang#80765 (resolve: Simplify collection of traits in scope) - rust-lang#80932 (Allow downloading LLVM on Windows and MacOS) - rust-lang#80983 (Remove is_dllimport_foreign_item definition from cg_ssa) - rust-lang#81064 (Support non-stage0 check) - rust-lang#81080 (Force vec![] to expression position only) - rust-lang#81082 (BTreeMap: clean up a few more comments) - rust-lang#81084 (Use Option::map instead of open-coding it) - rust-lang#81095 (Use Option::unwrap_or instead of open-coding it) - rust-lang#81107 (Add NonZeroUn::is_power_of_two) Failed merges: r? `@ghost` `@rustbot` modify labels: rollup

jyn514 added the S-experimental Status: Ongoing experiment that does not require reviewing and won't be merged in its current state. label Jan 12, 2021

rust-highfive assigned Mark-Simulacrum Jan 12, 2021

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jan 12, 2021

jyn514 force-pushed the download-windows-llvm branch from 13db0d3 to 98ef3a8 Compare January 12, 2021 01:37

This comment has been minimized.

Sign in to view

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 12, 2021

This comment has been minimized.

Sign in to view

jyn514 force-pushed the download-windows-llvm branch from 98ef3a8 to 71bd5d6 Compare January 12, 2021 01:46

This comment has been minimized.

Sign in to view

jyn514 added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jan 13, 2021

jyn514 changed the title ~~Allow downloading LLVM on Windows~~ Allow downloading LLVM on Windows and MacOS Jan 13, 2021

This comment has been minimized.

Sign in to view

jyn514 force-pushed the download-windows-llvm branch from 8dd7666 to 3bfa458 Compare January 13, 2021 14:45

nagisa reviewed Jan 13, 2021

View reviewed changes

src/bootstrap/bootstrap.py Show resolved Hide resolved

nagisa reviewed Jan 13, 2021

View reviewed changes

src/bootstrap/native.rs Show resolved Hide resolved

jyn514 force-pushed the download-windows-llvm branch from 3bfa458 to aad9437 Compare January 13, 2021 22:14

Mark-Simulacrum approved these changes Jan 16, 2021

View reviewed changes

src/bootstrap/config.rs Outdated Show resolved Hide resolved

src/bootstrap/dist.rs Outdated Show resolved Hide resolved

src/bootstrap/native.rs Show resolved Hide resolved

jyn514 added 2 commits January 15, 2021 22:07

Add all tier 1 platforms to supported platforms for "if-available"

5c4adbe

... and update the comment in `config.toml.example`

jyn514 force-pushed the download-windows-llvm branch from dd1f5dc to 5c4adbe Compare January 16, 2021 03:09

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 16, 2021

m-ou-se mentioned this pull request Jan 16, 2021

Rollup of 14 pull requests #81102

Closed

m-ou-se mentioned this pull request Jan 17, 2021

Rollup of 13 pull requests #81111

Closed

m-ou-se mentioned this pull request Jan 17, 2021

Rollup of 13 pull requests #81113

Merged

bors merged commit cfe2253 into rust-lang:master Jan 17, 2021

rustbot added this to the 1.51.0 milestone Jan 17, 2021

jyn514 deleted the download-windows-llvm branch January 17, 2021 19:34

jyn514 mentioned this pull request Jan 20, 2021

Remove "Skipping LLVM build" section rust-lang/rustc-dev-guide#1030

Merged

jyn514 mentioned this pull request Apr 27, 2022

Move download-ci-llvm out of bootstrap.py #95170

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow downloading LLVM on Windows and MacOS #80932

Allow downloading LLVM on Windows and MacOS #80932

jyn514 commented Jan 12, 2021 •

edited

Loading

rust-highfive commented Jan 12, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nagisa Jan 13, 2021 •

edited

Loading

jyn514 Jan 13, 2021 •

edited

Loading

Mark-Simulacrum commented Jan 16, 2021

jyn514 commented Jan 16, 2021

bors commented Jan 16, 2021

Allow downloading LLVM on Windows and MacOS #80932

Allow downloading LLVM on Windows and MacOS #80932

Conversation

jyn514 commented Jan 12, 2021 • edited Loading

Current Status

Windows

rust-highfive commented Jan 12, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nagisa Jan 13, 2021 • edited Loading

Choose a reason for hiding this comment

jyn514 Jan 13, 2021 • edited Loading

Choose a reason for hiding this comment

Mark-Simulacrum commented Jan 16, 2021

jyn514 commented Jan 16, 2021

bors commented Jan 16, 2021

jyn514 commented Jan 12, 2021 •

edited

Loading

nagisa Jan 13, 2021 •

edited

Loading

jyn514 Jan 13, 2021 •

edited

Loading