trans: Use LLVM's writeArchive to modify archives #26926
Conversation
|
r? @brson |
|
r? @huonw (rust_highfive has picked a reviewer for you, use r? to override) |
|
cc @Kingsquee |
|
I believe this should also remove the ability to specify an "ar" program in target specs? |
Indeed! I haven't done so here as I suspect we'll want to leave them around for awhile to work with 3.5/3.6 (and perhaps bugs that come up in the LLVM implementation). We'll also probably want to just accept them as an option for forever and perhaps start emitting a warning one day saying "you don't need to do this any more". |
|
Beautiful! |
|
@bors r+ |
|
📌 Commit 08847de has been approved by |
brson
added
the
relnotes
|
Does this bring us one step closer to not needing a bunch of flags for seamless cross compiling? (has it been tested for cross compilation?) |
|
@richo It should indeed, yes! I haven't tested this particular PR for cross compilation yet, but every time before I've used llvm-ar for cross compilation it works, so I suspect that this will also work. We also may have to turn this off by default and put it behind a |
|
@bors: r- Investigating some failures on OSX |
alexcrichton
force-pushed
the
llvm-archive-writer
branch
from
08847de
to
1c5c84f
Compare
|
re-r? @brson I've now discovered that there are multiple formats for archives (or at least various tweaks on flavors). The known formats (at least by LLVM) are these, and our current revision of LLVM can only emit archives in the GNU format. Support has been added very recently for BSD formats (what I believe OSX uses), but it's not present in our LLVM version currently and we can't upgrade due to some assorted test failures. I updated this to add an |
|
@bors r+ |
|
📌 Commit 1c5c84f has been approved by |
|
⌛ Testing commit 1c5c84f with merge 7b855c0... |
|
💔 Test failed - auto-mac-64-opt |
alexcrichton
force-pushed
the
llvm-archive-writer
branch
from
1c5c84f
to
e5ebaa6
Compare
We have previously always relied upon an external tool, `ar`, to modify archives that the compiler produces (staticlibs, rlibs, etc). This approach, however, has a number of downsides: * Spawning a process is relatively expensive for small compilations * Encoding arguments across process boundaries often incurs unnecessary overhead or lossiness. For example `ar` has a tough time dealing with files that have the same name in archives, and the compiler copies many files around to ensure they can be passed to `ar` in a reasonable fashion. * Most `ar` programs found do **not** have the ability to target arbitrary platforms, so this is an extra tool which needs to be found/specified when cross compiling. The LLVM project has had a tool called `llvm-ar` for quite some time now, but it wasn't available in the standard LLVM libraries (it was just a standalone program). Recently, however, in LLVM 3.7, this functionality has been moved to a library and is now accessible by consumers of LLVM via the `writeArchive` function. This commit migrates our archive bindings to no longer invoke `ar` by default but instead make a library call to LLVM to do various operations. This solves all of the downsides listed above: * Archive management is now much faster, for example creating a "hello world" staticlib is now 6x faster (50ms => 8ms). Linking dynamic libraries also recently started requiring modification of rlibs, and linking a hello world dynamic library is now 2x faster. * The compiler is now one step closer to "hassle free" cross compilation because no external tool is needed for managing archives, LLVM does the right thing! This commit does not remove support for calling a system `ar` utility currently. We will continue to maintain compatibility with LLVM 3.5 and 3.6 looking forward (so the system LLVM can be used wherever possible), and in these cases we must shell out to a system utility. All nightly builds of Rust, however, will stop needing a system `ar`.
alexcrichton
force-pushed
the
llvm-archive-writer
branch
from
e5ebaa6
to
4a82427
Compare
We have previously always relied upon an external tool, `ar`, to modify archives that the compiler produces (staticlibs, rlibs, etc). This approach, however, has a number of downsides: * Spawning a process is relatively expensive for small compilations * Encoding arguments across process boundaries often incurs unnecessary overhead or lossiness. For example `ar` has a tough time dealing with files that have the same name in archives, and the compiler copies many files around to ensure they can be passed to `ar` in a reasonable fashion. * Most `ar` programs found do **not** have the ability to target arbitrary platforms, so this is an extra tool which needs to be found/specified when cross compiling. The LLVM project has had a tool called `llvm-ar` for quite some time now, but it wasn't available in the standard LLVM libraries (it was just a standalone program). Recently, however, in LLVM 3.7, this functionality has been moved to a library and is now accessible by consumers of LLVM via the `writeArchive` function. This commit migrates our archive bindings to no longer invoke `ar` by default but instead make a library call to LLVM to do various operations. This solves all of the downsides listed above: * Archive management is now much faster, for example creating a "hello world" staticlib is now 6x faster (50ms => 8ms). Linking dynamic libraries also recently started requiring modification of rlibs, and linking a hello world dynamic library is now 2x faster. * The compiler is now one step closer to "hassle free" cross compilation because no external tool is needed for managing archives, LLVM does the right thing! This commit does not remove support for calling a system `ar` utility currently. We will continue to maintain compatibility with LLVM 3.5 and 3.6 looking forward (so the system LLVM can be used wherever possible), and in these cases we must shell out to a system utility. All nightly builds of Rust, however, will stop needing a system `ar`.
…lexcrichton Document that `-C ar=PATH` doesn't do anything Are there any plans to use an external archiver in the future? IIRC, it was used before, but its use was replaced with LLVM's built-in archive management machinery. I can't found a relevant PR though. EDIT: Found it - rust-lang#26926! The `-C` option is stable so it still can't be removed right away even if there are no plans to use it (but maybe it can be deprecated?). Target specifications have a field for archiver as well, which is unused too (these ones are unstable, so I guess it can be removed). r? @alexcrichton
We have previously always relied upon an external tool,
ar, to modify archivesthat the compiler produces (staticlibs, rlibs, etc). This approach, however, has
a number of downsides:
or lossiness. For example
arhas a tough time dealing with files that havethe same name in archives, and the compiler copies many files around to ensure
they can be passed to
arin a reasonable fashion.arprograms found do not have the ability to target arbitraryplatforms, so this is an extra tool which needs to be found/specified when
cross compiling.
The LLVM project has had a tool called
llvm-arfor quite some time now, but itwasn't available in the standard LLVM libraries (it was just a standalone
program). Recently, however, in LLVM 3.7, this functionality has been moved to a
library and is now accessible by consumers of LLVM via the
writeArchivefunction.
This commit migrates our archive bindings to no longer invoke
arby defaultbut instead make a library call to LLVM to do various operations. This solves
all of the downsides listed above:
staticlib is now 6x faster (50ms => 8ms). Linking dynamic libraries also
recently started requiring modification of rlibs, and linking a hello world
dynamic library is now 2x faster.
no external tool is needed for managing archives, LLVM does the right thing!
This commit does not remove support for calling a system
arutility currently.We will continue to maintain compatibility with LLVM 3.5 and 3.6 looking forward
(so the system LLVM can be used wherever possible), and in these cases we must
shell out to a system utility. All nightly builds of Rust, however, will stop
needing a system
ar.