Store LLVM bitcode in object files, not compressed #71528

alexcrichton · 2020-04-24T19:25:10Z

This commit is an attempted resurrection of #70458 where LLVM bitcode
emitted by rustc into rlibs is stored into object file sections rather
than in a separate file. The main rationale for doing this is that when
rustc emits bitcode it will no longer use a custom compression scheme
which makes it both easier to interoperate with existing tools and also
cuts down on compile time since this compression isn't happening.

The blocker for this in #70458 turned out to be that native linkers
didn't handle the new sections well, causing the sections to either
trigger bugs in the linker or actually end up in the final linked
artifact. This commit attempts to address these issues by ensuring that
native linkers ignore the new sections by inserting custom flags with
module-level inline assembly.

Note that this does not currently change the API of the compiler at all.
The pre-existing -C bitcode-in-rlib flag is co-opted to indicate
whether the bitcode should be present in the object file or not.

Finally, note that an important consequence of this commit, which is also
one of its primary purposes, is to enable rustc's -Clto bitcode
loading to load rlibs produced with -Clinker-plugin-lto. The goal here
is that when you're building with LTO Cargo will tell rustc to skip
codegen of all intermediate crates and only generate LLVM IR. Today
rustc will generate both object code and LLVM IR, but the object code is
later simply thrown away, wastefully.

rust-highfive · 2020-04-24T19:25:14Z

r? @ecstatic-morse

(rust_highfive has picked a reviewer for you, use r? to override)

rust-highfive · 2020-04-24T19:25:15Z

⚠️ Warning ⚠️

These commits modify submodules.

alexcrichton · 2020-04-24T19:25:51Z

As an extra note, I've tested this on Windows/mac/linux and confirmed that where bytecode sections previously appeared they're now all gone by default. Additionally I've tested that the builders which originally failed on #70458 pass with this implementation.

petrochenkov · 2020-04-24T19:51:44Z

Nit: I'm not sure why the option name changed since it was approved in #70458.
Is there case for emitting bitcode into other library kinds in the future making the -in-rlib part obsolete?

alexcrichton · 2020-04-24T20:04:00Z

I'm not really sure, most of this bytecode embedding stuff has been driven by demand and it's hard to predict future demands. It's also not on stable yet, so if we want to switch up the name that's also fine. Given the fall out in #70458 though I would personally prefer to leave refactorings/etc to a future PR to make sure this actually can land first (and doesn't need to be backed out)

nnethercote

Thanks for doing this!

#70458 had a commit "Don't copy bytecode files into the incr. comp. cache." I think that could be applied here?

Can the -Zembed-bitcode flag be removed now? Perhaps in a follow-up?

src/librustc_codegen_llvm/back/write.rs

alexcrichton · 2020-04-27T14:55:05Z

Ah yeah I saw the extra patch, although I'm personally not very familiar with it. I was also hoping that given the hiccups we saw earlier we probably don't want to design too much around this until we're more sure that it works. WDYT about landing this first, making sure it doesn't wildly break everyone on nightly, and then we can land some cleanups which this unlocks?

ecstatic-morse · 2020-04-27T16:34:11Z

r? @nnethercote

bjorn3 · 2020-04-27T17:48:44Z

src/librustc_codegen_llvm/back/archive.rs

@@ -129,8 +129,8 @@ impl<'a> ArchiveBuilder<'a> for LlvmArchiveBuilder<'a> {
        let obj_start = name.to_owned();

        self.add_archive(rlib, move |fname: &str| {
-            // Ignore bytecode/metadata files, no matter the name.
-            if fname.ends_with(RLIB_BYTECODE_EXTENSION) || fname == METADATA_FILENAME {
+            // Ignore metadata files, no matter the name.


Suggested change

// Ignore metadata files, no matter the name.

// Ignore metadata files

bjorn3 · 2020-04-27T17:53:01Z

src/librustc_codegen_ssa/back/link.rs

@@ -1829,7 +1811,7 @@ fn add_upstream_rust_crates<'a, B: ArchiveBuilder<'a>>(

            let mut any_objects = false;
            for f in archive.src_files() {
-                if f.ends_with(RLIB_BYTECODE_EXTENSION) || f == METADATA_FILENAME {
+                if f == METADATA_FILENAME {


Yay, one less reason to rewrite rlibs every time we link. I wonder if there is a way for the linker to skip the metadata file without having to remove it from the archive.

That is not true, rlibs are already passed directly to the linker. Only in situations like LTO where we want to remove object code or we're doing something -force_load do we remove things from rlibs.

Missed the return above.

nnethercote · 2020-04-27T21:10:46Z

WDYT about landing this first, making sure it doesn't wildly break everyone on nightly, and then we can land some cleanups which this unlocks?

Sounds good. That way we can also see if the extra patch has any measurable perf impact. I will review in a bit.

src/librustc_codegen_llvm/back/lto.rs

src/test/ui/lto-rustc-loads-linker-plugin.rs

src/test/ui/lto-thin-rustc-loads-linker-plugin.rs

src/librustc_codegen_ssa/back/write.rs

nnethercote · 2020-04-27T23:14:53Z

One more thing: src/doc/rustc/src/codegen-options/index.md mentions compressed bitcode for the bitcode-in-rlib option, please update that. (And a gold star if you could move that section up near the top so it's in alphabetical order; I missed doing that in #71374.)

alexcrichton · 2020-04-28T18:45:45Z

Ok updated!

nnethercote · 2020-04-28T21:09:30Z

Follow-ups required:

Possibly rename -Cbitcode-in-rlib as -Cembed-bitcode (and removed -Zembed-bitcode). This will require backporting this to beta?
The "Don't copy bytecode files into the incr. comp. cache." commit from Move LLVM bitcode destination #70458.

@bors r+

bors · 2020-04-28T21:09:31Z

📌 Commit 95fd968d9c95e1e5d872f2f0d5378f38ff658f86 has been approved by nnethercote

nnethercote · 2020-04-28T23:55:39Z

@bors p=1

Because follow-ups to beta will be necessary if this lands successfully.

bors · 2020-04-29T23:47:33Z

⌛ Testing commit ef89cc8 with merge 1357af3...

bors · 2020-04-30T03:04:19Z

☀️ Test successful - checks-azure
Approved by: nnethercote
Pushing 1357af3 to master...

rust-highfive · 2020-04-30T03:05:33Z

📣 Toolstate changed by #71528!

Tested on commit 1357af3.
Direct link to PR: #71528

🎉 rustc-dev-guide on linux: test-fail → test-pass (cc @JohnTitor @amanjeev @spastorino @mark-i-m).

@JohnTitor

Tested on commit rust-lang/rust@1357af3. Direct link to PR: <rust-lang/rust#71528> 🎉 rustc-dev-guide on linux: test-fail → test-pass (cc @JohnTitor @amanjeev @spastorino @mark-i-m).

This commit finishes work first pioneered in rust-lang#70458 and started in rust-lang#71528. The `-C bitcode-in-rlib` option, which has not yet reached stable, is renamed to `-C embed-bitcode` since that more accurately reflects what it does now anyway. Various tests and such are updated along the way as well. This'll also need to be backported to the beta channel to ensure we don't accidentally stabilize `-Cbitcode-in-rlib` as well.

…thercote Rename `bitcode-in-rlib` option to `embed-bitcode` This commit finishes work first pioneered in rust-lang#70458 and started in rust-lang#71528. The `-C bitcode-in-rlib` option, which has not yet reached stable, is renamed to `-C embed-bitcode` since that more accurately reflects what it does now anyway. Various tests and such are updated along the way as well. This'll also need to be backported to the beta channel to ensure we don't accidentally stabilize `-Cbitcode-in-rlib` as well.

@nnethercote

…nethercote Don't copy bytecode files into the incr. comp. cache. It's no longer necessary now that bitcode is embedded into object files. This change meant that `WorkProductFileKind::Bytecode` is no longer necessary, which means that type is no longer necessary, which allowed several places in the code to become simpler. This commit was written by @nnethercote in rust-lang#70458 but that didn't land. In the meantime though we managed to land it in rust-lang#71528 and that doesn't seem to be causing too many fires, so I'm re-sending this patch!

This commit updates Cargo's implementation of LTO builds to do less work and hopefully be faster when doing a cold build. Additionaly this should save space on disk! The general idea is that the compiler does not need object files if it's only going to perform LTO with some artifacts. In this case all rustc needs to do is load bitcode from dependencies. This means that if you're doing an LTO build generating object code for intermediate dependencies is just wasted time! Here Cargo is updated with more intrusive knowledge about LTO. Cargo will now analyze the dependency graph to figure out which crates are being compiled with LTO, and then it will figure out which dependencies only need to have bitcode in them. Pure-bitcode artifacts are emitted with the `-Clinker-plugin-lto` flag. Some artifacts are still used in multiple scenarios (such as those shared between build scripts and final artifacts), so those are not compiled with `-Clinker-plugin-lto` since the linker is not guaranteed to know how to perform LTO. This functionality was recently implemented in rust-lang/rust#71528 where rustc is now capable of reading bitcode from `-Clinker-plugin-lto` rlibs. Previously rustc would only read its own format of bitcode, but this has now been extended! This support is now on nightly, hence this PR.

Don't force rustc to do codegen for LTO builds This commit updates Cargo's implementation of LTO builds to do less work and hopefully be faster when doing a cold build. Additionaly this should save space on disk! The general idea is that the compiler does not need object files if it's only going to perform LTO with some artifacts. In this case all rustc needs to do is load bitcode from dependencies. This means that if you're doing an LTO build generating object code for intermediate dependencies is just wasted time! Here Cargo is updated with more intrusive knowledge about LTO. Cargo will now analyze the dependency graph to figure out which crates are being compiled with LTO, and then it will figure out which dependencies only need to have bitcode in them. Pure-bitcode artifacts are emitted with the `-Clinker-plugin-lto` flag. Some artifacts are still used in multiple scenarios (such as those shared between build scripts and final artifacts), so those are not compiled with `-Clinker-plugin-lto` since the linker is not guaranteed to know how to perform LTO. This functionality was recently implemented in rust-lang/rust#71528 where rustc is now capable of reading bitcode from `-Clinker-plugin-lto` rlibs. Previously rustc would only read its own format of bitcode, but this has now been extended! This support is now on nightly, hence this PR.

rust-highfive assigned ecstatic-morse Apr 24, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 24, 2020

alexcrichton mentioned this pull request Apr 24, 2020

Move LLVM bitcode destination #70458

Closed

alexcrichton requested a review from nnethercote April 24, 2020 19:35

nnethercote reviewed Apr 24, 2020

View reviewed changes

src/librustc_codegen_llvm/back/write.rs Outdated Show resolved Hide resolved

alexcrichton force-pushed the no-more-bitcode branch from 13852a7 to ae4ebb0 Compare April 27, 2020 14:54

alexcrichton force-pushed the no-more-bitcode branch from ae4ebb0 to 8dfceec Compare April 27, 2020 15:09

rust-highfive assigned nnethercote and unassigned ecstatic-morse Apr 27, 2020

bjorn3 reviewed Apr 27, 2020

View reviewed changes

nnethercote reviewed Apr 27, 2020

View reviewed changes

alexcrichton force-pushed the no-more-bitcode branch 2 times, most recently from da0af77 to 95fd968 Compare April 28, 2020 18:45

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 28, 2020

alexcrichton mentioned this pull request Apr 29, 2020

Upcoming changes around -Zembed-bitcode getditto/rust-bitcode#9

Closed

bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 30, 2020

bors merged commit 1357af3 into rust-lang:master Apr 30, 2020

This was referenced Apr 30, 2020

upgrade chalk and use chalk-solve/chalk-ir/chalk-rust-ir #69406

Merged

Implement new asm! syntax from RFC 2850 #69171

Merged

alexcrichton deleted the no-more-bitcode branch April 30, 2020 14:28

alexcrichton mentioned this pull request Apr 30, 2020

Rename bitcode-in-rlib option to embed-bitcode #71716

Merged

This was referenced May 1, 2020

Don't force rustc to do codegen for LTO builds rust-lang/cargo#8192

Merged

Don't copy bytecode files into the incr. comp. cache. #71754

Merged

IsaacWoods mentioned this pull request May 4, 2020

Build errors on recent nightlies when compiling rustc-std-workspace-core on UEFI targets #71880

Closed

nnethercote mentioned this pull request May 6, 2020

Building with LTO should skip "compilation" #43212

Closed

jschwe mentioned this pull request May 7, 2020

LTO doesn't work with rust nightlies after 2020-04-29 hermit-os/hermit-rs#15

Closed

phil-opp mentioned this pull request May 11, 2020

error: failed to get bitcode from object file for LTO rust-osdev/cargo-xbuild#69

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store LLVM bitcode in object files, not compressed #71528

Store LLVM bitcode in object files, not compressed #71528

alexcrichton commented Apr 24, 2020

rust-highfive commented Apr 24, 2020

rust-highfive commented Apr 24, 2020

alexcrichton commented Apr 24, 2020

petrochenkov commented Apr 24, 2020

alexcrichton commented Apr 24, 2020

nnethercote left a comment

alexcrichton commented Apr 27, 2020

ecstatic-morse commented Apr 27, 2020

bjorn3 Apr 27, 2020

bjorn3 Apr 27, 2020

alexcrichton Apr 27, 2020

bjorn3 Apr 27, 2020

nnethercote commented Apr 27, 2020

nnethercote commented Apr 27, 2020

alexcrichton commented Apr 28, 2020

nnethercote commented Apr 28, 2020

bors commented Apr 28, 2020

nnethercote commented Apr 28, 2020

bors commented Apr 29, 2020

bors commented Apr 30, 2020

rust-highfive commented Apr 30, 2020

	// Ignore metadata files, no matter the name.
	// Ignore metadata files

Store LLVM bitcode in object files, not compressed #71528

Store LLVM bitcode in object files, not compressed #71528

Conversation

alexcrichton commented Apr 24, 2020

rust-highfive commented Apr 24, 2020

rust-highfive commented Apr 24, 2020

alexcrichton commented Apr 24, 2020

petrochenkov commented Apr 24, 2020

alexcrichton commented Apr 24, 2020

nnethercote left a comment

Choose a reason for hiding this comment

alexcrichton commented Apr 27, 2020

ecstatic-morse commented Apr 27, 2020

bjorn3 Apr 27, 2020

Choose a reason for hiding this comment

bjorn3 Apr 27, 2020

Choose a reason for hiding this comment

alexcrichton Apr 27, 2020

Choose a reason for hiding this comment

bjorn3 Apr 27, 2020

Choose a reason for hiding this comment

nnethercote commented Apr 27, 2020

nnethercote commented Apr 27, 2020

alexcrichton commented Apr 28, 2020

nnethercote commented Apr 28, 2020

bors commented Apr 28, 2020

nnethercote commented Apr 28, 2020

bors commented Apr 29, 2020

bors commented Apr 30, 2020

rust-highfive commented Apr 30, 2020