Add a Cargo-based build system to replace the Makefiles #27003
Conversation
|
Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon. If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. The way Github handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes. Please see the contribution instructions for more information. |
This commit adds a Cargo-based build system, in parallel to the old
Makefile-based build system. The new Cargo-based build system is supposed
to replace the old system, and the old system will be marked for deletion
once the new system has been tested thoroughly.
On a high-level, this commit (and the commits to follow in this pull request):
1. Add the necessary `Cargo.toml` files to convert the crates into Cargo
packages.
2. Add a `build.rs` script when building native runtime libraries is
required.
3. Add a Python script `etc/build_llvm.py` to invoke the LLVM build system.
4. Add a driver script `etc/build_rust.py` to parse the command line
arguments, build LLVM, and invoke Cargo to bootstrap the compiler,
build the docs, and build and run the compiler and library tests.
With the new Cargo-based build system we no longer have Makefile, which is used by the fds-are-cloexec test. This commit makes said test use README.md instead.
This is necessary for librustc_bitflags to compile under Mac OS X in a Cargo-based build.
This commit adds `#[allow]` attributes to liballoc, libcollections, libcollectionstest, libcoretest, librand, and libstd to silence unnecessary warnings when they are compiled under `cargo test`.
Update .gitignore to include files produced by Cargo-based builds.
Cargo puts the generated files under $OUT_DIR. We update the source to look at $OUT_DIR when compiling under Cargo.
The old build system does not run doctests. Cargo runs them by default.
This is necessary to build compiler-rt under Windows, with the standalone mingw distribution.
| @@ -0,0 +1,15 @@ | |||
| // Copyright 2012 The Rust Project Developers. See the COPYRIGHT | |||
There was a problem hiding this comment.
I’m wondering what’s the reason for all these wrapper files. Can’t it live in src/librustc_driver?
Also the copyright year is quite far in the past.
There was a problem hiding this comment.
Can’t it live in src/librustc_driver?
This way one can simply invoke cargo build on one crate (driver) and it will make cargo build both rustc and rustdoc.
I’m not entirely sure about it, but AFAIR on most systems dynamic linking will also help OS to not load the same code for every running process. It would not be too surprising to see multiple instances of rustc running concurrently, hence increased resident memory usage. |
|
@nagisa Because the OS is loading the same What dynamic linking gives you is the ability to share common libraries between Incidentally this is also what LLVM does by default --- the build system will link the |
Because some system may have `python == python3`
| /src/*/target | ||
| /src/*/tmp | ||
| /target/ | ||
| !/src/librustc_back/target |
There was a problem hiding this comment.
This can probably be refactored to just:
Cargo.lock
target
!src/librustc_back/target
| } else if artifacts_dir.join("bin").join("llc.exe").is_file() { | ||
| return artifacts_dir; | ||
| } | ||
| } |
There was a problem hiding this comment.
Unfortunately this doesn't work well if you've built LLVM twice (e.g. once with asserts and once without) as it'll just pick whatever's first in the vector, not the current build.
There are concerns with ABI compatibility between stages that force our hand in compiling the compiler 3 times. I believe this will still need to compile 3 compilers even though we have a full nightly available. Basically architecturally the bootstrap needs to stay relatively the same to ensure that plugins work in stage2.
I have expected this in the past to break plugins, did the plugin tests run successfully for you?
Can we hold off on LTO for now?
It should be possible to pass this via
Yeah we use this daily for building nightlies so it'll need to be covered before we move off makefiles.
I'll comment more over there, but I think that we'll not want to have to work around this, I expect the output of each stage to be in a separate location. |
|
Ok @brson and I had a chat about this yesterday, and the conclusion was somewhat indecisive. We agree that we probably want to eventually move in the direction of a Cargo-based build system in the future, but it's somewhat unclear how immediately we can do so. It'd be unfortunate to have to maintain two build systems, so once we land this we definitely want to have the makefiles basically immediately on their way out (assuming we have to have a small transitionary period at least). There's also a pretty huge amount of logic in the makefiles which isn't currently duplicated into the python script, for example:
This is somewhat ok for a first pass, but it's unclear to what degree this should be dealt with before landing. Unfortunately @brson and I won't be able to spend a large amount of time on this in the immediate future, so this would have to be a mostly community-maintained build system for Rust in the near future. It'd be a shame for this to land, find a number of bugs, and then bitrot in-tree until we finally remove it as it's too broken. Overall we think we're willing to land this in-tree in a relatively conservative fashion, but we'd like to make sure that we're set to have a clear path forward with the implementation. We can help out with reviews for the time being but we won't be able to make major contributions until perhaps later this year. I think a great first step would be to identify any area that Cargo is lacking that needs to be improved perhaps before landing this in tree. I suspect that Cargo as-is right this red-hot-second may not be sufficient for all our build needs, but with a few tweaks I'm sure it can be! |
|
@alexcrichton Yeah I definitely did hit a few bugs/limitations in Cargo when I was working on this. Some of them can be worked around but they should be addressed properly before merging this in tree. I'm going to close this for now. |
|
Ok, I'd love to stay posted on this though, and feel free to ping me with any questions! |
|
Would it be possible (presumingly after adding this cargo-based build system) to replace Makefiles with Ninja build files? |
|
@boghison presumably, makefiles would be removed after such a change, and there would be no place for other build systems. |
This pull request adds a Cargo-based build system, in parallel to the old Makefile-based one. The two build systems should be able to function alongside each other, but it is preferred to use the new one and the Makefiles will be deleted once the new system has been thoroughly tested and is considered mature enough.
On a high-level, this pull request:
Cargo.tomlfiles to convert the crates into Cargo packages.etc/build_llvm.pyto drive the LLVM build system.etc/build_rust.pyto drive the whole system. This driver script will parse the command line arguments, build LLVM, and then invoke Cargo to bootstrap the compiler, build the docs, and build and run the compiler tests and library tests.The new build system offers a number of advantages over the old one. They include:
cargo build --target=<target-triple>libstdtests without bootstrapping the compiler. This can save quite some time when you are developing the standard library.makeand other Unix tools when they are unavailable. You can now build the compiler under a standard Windows command line window, without having MSYS2 installed (see the list in the tracking Windows issue).There are a number of technical differences between the two build systems:
rustcandcargo). Because the stage0 compiler is already a hosted compiler (instead of a freestanding one), there is no need to go through the staged api and compile a hosted stage0 compiler. Instead we will simply use nightly to compile the stage1 compiler and then use the stage1 compiler to compile the stage2 compiler.rustcandrustdoc) will be statically linked against the standard library and compiler crates. This is the default behaviour of Cargo. It also makes it easier to redistribute the compiler as there won't be any dylib dependencies of the executables, although it doesn't really make the distribution smaller (they have roughly the same size on release builds as I measured).opt-level=3andlto=true, to make them faster.compiler-rt, we will not use the builtin Makefiles and instead invoke the C toolchain directly. This is much simpler and works on platforms where gnumakeis not universally available. This requires an update to thecompiler-rtsource by adding a small header file which is missing for Windows (it is available for Linux and darwin), because the standalone mingw doesn't include it.Current limitations of the new system:
-C link-args=-DEF:to rustc. This requires extending Cargo and support for MSVC will come in pull requests to follow.cargo build --target=<target-triple>, and the code is written to handle cross-bootstrapping properly, in practice there will be platform differences. Therefore cross-bootstrapping needs a lot more testing than it currently has.To test, one needs to
Pull Include path to rustc when generating package metadata cargo#1802 and recompile Cargo.
You will also need to pull Add minimal Windows SDK to build with standalone mingw64 compiler-rt#9 if you want to build on Windows.
On a Unix-based system, run
On Windows, open a command line window and run
To see a list of supported options, pass
--helpto the build script. For instance, you might want to use--enable-debugto build with debug profile (the default is to build with the release profile).See also the discussion in #26493.
r? @alexcrichton