Add an alternative build system written in Rust

[cl91]

This pull request adds an alternative build system written in Rust. This adds a Cargo package, build-rust, which will be compiled (by Cargo) before the build system itself runs throught the build process for bootstrapping the compiler.

The primary reason for this new build system, apart from being much easier to understand and to hack than the old one, is to enable building Rust under Windows outside a MSYS shell. This implements one of the items in the tracking Windows issue.

With the new build system, the only dependencies that one needs in order to compile Rust (on any platform) are:

• cmake and python2: for building LLVM
• git: for cloning the repo
• curl, tar, and bzip2: for downloading and unpacking the stage0 snapshot
• Appropriate C/C++ toolchain (such as gcc)

This is tested under Windows (win7-x64-mingw64), Linux (x86_64-gcc-5.0) and Darwin (x86_64-clang-3.6). In all cases the build system is able to build LLVM and all runtime libraries and bootstrap a working stage2 compiler from the stage0 snapshot.

This is only the first step towards a fully-functional build system for the Rust compiler. Below is a list of missing features/functions in the build system as it currently stands:

• MSVC build doesn't work currently as compiler-rt cannot be compiled by MSVC (cl is missing a few important C99/C11 features such as _Atomic and _Complex). One solution is to use mingw (or gcc) to compile compiler-rt.
• The build system doesn't build docs nor run compiler tests. This can be added quite easily.
• Cross-bootstrapping is not tested and will not work.
• The build system can only run parallel jobs for building LLVM and external libraries. It cannot parallelize building of the rustc compiler. To do this reliably one would need the dependency information on the various librustc and libstd crates. This is best done by converting them into Cargo packages and invoke Cargo to build them.

@retep998

r? @alexcrichton

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. The way Github handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[steveklabnik]

there should be a space after this paragraph and before the triple backticks, same everywhere else

also, a : at the end of this line would be nice, keep it consistent with the others

[cl91]

Good call. Fixed.

[steveklabnik]

neat! :

[retep998]

Whoa.

[petrochenkov]

Wow!
I expected that item in rust-lang/rfcs#1061 to be resolved with good old CMake (+Ninja), like LLVM does.

[alexcrichton]

Whoa! @brson and I have long wanted to do something like this, and it's quite interesting to see this PR!

Before I go too much into reviewing this, I wanted to talk a bit about the high-level points here. The motivation for doing this, building on Windows without MinGW, is certainly quite admirable, but this is quite a heavy hammer to fix that problem. Currently we've got MSVC builds working from an MSYS shell, and while awkward, it's only required for building Rust itself, not Rust projects at large (thanks to Cargo). That being said, however, the point of hacking on the build system most certainly rings true to me as the Makefiles are impenetrable.

Overall, I don't think that we want to plan on maintaining two build systems in this repository. I personally would be quite hesitant to have an unsupported build system in the repo (but one official one on the side). The two will inevitably drift apart and it'll be unclear of which one should be used over the other. If we want to add a new build system then we'll definitely want to have both in-tree for some time (to debug and have an easy fallback if it doesn't work), but the current makefiles would definitely be slated for deletion at that point.

In terms of architecture, this will also be the second major build system for building Rust code written in Rust (the first being Cargo itself), and it's somewhat unfortunate to have this duplication. The limitations you've listed of lacking documentation, testing, cross compiling, and parallel builds are often fundamental changes (or so I've found through Cargo) and can significantly complicate the build system. We've typically thought in the past that if we're going to pursue this that we'll try to use Cargo itself to build the code in this repository so we wouldn't have to maintain two separate systems for building Rust code.

Coincidentally, I've been dabbling recently in building the compiler with Cargo (much of the work inspired by @japaric's std-with-cargo project), and you can see my branch for these changes as well. I will probably not be actively pursuing this any time soon, however (as in I won't be stepping on your toes any time soon)

---

In summary, I think there's a few fundamental drawbacks of this patch in its current form that we may want to resolve before merging:

• Having a duplicate build system I believe will only work out if we plan on deleting the original build system ASAP.
• Having a duplicate build system for Rust code (the other being Cargo) will likely just end up with the two projects just converging one day so it may be best to work with Cargo itself

Some more minor points I'm also concerned about are:

• It doesn't look like there's an automated way to download a Cargo binary to run the build_rust executable
• The lack of features like parallel compilation, cross compilation, tests, and documentation will prevent this from replacing the current build system.

Despite all this, however, I would absolutely love to help push this through! We stand to recover a significant amount of technical debt through using Cargo:

• No more Makefile madness -- aka hacking the "build system" will be much easier.
• MSVC can be supported as make and other Unix utilities won't be necessary.
• Leveraging a standard tool for building Rust code to build the compiler will only inevitably improve Cargo itself
• Leveraging Cargo will easily allow crates to move in and out of the Rust repository as well as allowing rustc to depend on crates.io (or publish these crates on crates.io)
• Using build scripts to handle native dependencies will greatly modularize how they're built and make it very easy to script and/or modify their builds.
• Cross compiling the compiler will be much easier

And I'm sure I'm missing a few! These benefits are so nice to have that I'm more than willing to help out in any way with this!

[cl91]

Yes these are good points. I agree this may not be in a state to be merged as it lacks several useful features and this adds maintenance burdens to support two build systems while one matures.

Part of my original plan was to re-use Cargo as much as possible (especially when it comes to parallel jobs as dependencies will be taken care of), though I haven't started implementing this as there are non-trivial steps, such as building LLVM and runtime libraries, which are outside of Cargo's scope (also the problem of bootstrapping which I'm not sure if Cargo handles this out-of-box). Therefore I wanted to look into these issues first and to make sure that they are not tied to the current Makefile-based system in any fundamental way, and that it is indeed possible to implement a standalone build system that does not rely on Makefiles at all.

I'm closing this for now and will reopen/resubmit.

[eddyb]

@cl91 I think it might be possible to use cargo's own snapshots for bootstrapping rustc (without building cargo at all in the process, just using the snapshot).
Building native dependencies is usually done by a cargo build script, which is what your build_rust tool looks like (though your code does more than a simple build script).

[alexcrichton]

@cl91 ok, and feel free to ask me any questions! At a high level, here's what I'm thinking:

• We're going to have a top-level script driving Cargo, probably written in python. This script will handle cross compilation (how to call cargo) and how to call cargo in parallel.
• LLVM will be built ahead of time (not as part of a build script) by the driver script because some of the tools in the standard library and such need to use the LLVM tools (e.g. llc) to generate an object. This also makes building rustc_llvm somewhat reasonable.
• We'll use Cargo and Rust nightlies (instead of snapshots) as @eddyb mentioned

Depending on how complicated the driver script is we could write it in Rust (and just have a tiny amount of python to call that Rust and run it), but having a somewhat significant portion of the build system written in Python isn't necessarily the end of the world!

[cl91]

Depending on how complicated the driver script is we could write it in Rust

Yes this is what this patch currently does (to a large extend). It parses command line arguments, checks build prerequisites, invokes cmake to configure and build LLVM, and invokes c/c++ toolchain to build runtime libraries, etc. (the last one should probably be converted into a Cargo build script once we cargo-ify the crates). So I've got the driver script part basically sorted out.

After finishing all these steps the script simply invokes rustc to compile all the crates. That's why it can't do parallel builds because dependency information is missing. By cargo-ifying the crates we can have cargo do all the heavylifting of computing dependencies/passing the right arguments to rustc/etc... This is what I'll be looking into next.

This patch builds with Rust 1.0, so we shouldn't need nightly to run the driver script itself (we still need nightly rustc/cargo for compiling the crates).

[cl91]

@alexcrichton Hi Alex. Is there any way to specify the path where Cargo finds the rustc and rustdoc executable? A cursory look at the Cargo source code suggests that it doesn't. To compile the stage1 library from the stage0 compiler, one needs to tell Cargo where the newly-built rustc executable is. I don't think this is possible in Cargo currently?

[eddyb]

@cl91 You could modify PATH to do it - just make sure it's before everything else, so that a system-wide rustc installation would not get picked up by mistake.

[cl91]

@eddyb Right that should do the trick.

[alexcrichton]

@cl91, @eddyb, Cargo recently gained the ability to read the RUSTC environment variable or build.rustc configuration option to indicate where the compiler is.

[cl91]

@alexcrichton Having successfully compiled a working rustc and rustdoc from a Cargo-based build I'm now trying to bring the tests to use cargo. Running cargo test --manifest-path=src/libstd/Cargo.toml I am getting lots of E0152 errors from rustc. It appears that crate std is being linked twice by rustc causing it to complain about multiple definitions of lang-items. Is there any way to tell rustc to not link to std when compiling tests?

The errors that I am getting are

src/libstd/lib.rs:1:1: 1:1 error: duplicate entry for `const_ptr` [E0152]
src/libstd/lib.rs:1 // Copyright 2012-2014 The Rust Project Developers. See the COPYRIGHT
                    ^
src/libstd/lib.rs:1:1: 1:1 help: run `rustc --explain E0152` to see a detailed explanation
src/libstd/lib.rs:1:1: 1:1 error: duplicate entry for `mut_ptr` [E0152]
src/libstd/lib.rs:1 // Copyright 2012-2014 The Rust Project Developers. See the COPYRIGHT
                    ^
src/libstd/lib.rs:1:1: 1:1 help: run `rustc --explain E0152` to see a detailed explanation
src/libstd/lib.rs:1:1: 1:1 error: duplicate entry for `i8` [E0152]

and so on.

[alexcrichton]

@cl91 I didn't get around to poking around tests that much, so I'm not entirely sure where that's coming from. Do you have a branch I could poke around with? It would also be helpful to have a full log of cargo test -v --manifest-path ...

[cl91]

Do you have a branch I could poke around with?

@alexcrichton Yes. Checkout this branch https://github.com/cl91/rust/tree/cargoify.

To test, just run ./build.sh. It will invoke a small driver script (in src/etc/build_rust.py) to build LLVM, and then invoke cargo build to build the compiler (and rustdoc), and then invoke cargo test to try to run the libstd tests, at which point it fails.

I've tested it under Linux, Windows (mingw), and Mac and building the compiler should work fine.

Running ./build.sh gives a list of optional arguments the script accepts. You may want to use --llvm-root to specify an external LLVM root if you don't want to rebuild LLVM. Adding --verbose will pass the --verbose argument to Cargo.

The full list of error output when running the libstd tests can be found here: https://gist.github.com/cl91/f1019f0ab05ed22d6abf

[alexcrichton]

Ah ok I think I see what's going on here. The stdtest binary is linking to libtest from the compiler we just built, which is transitively bringing in that libcore. The standard library is also linking to its own copy of libcore, however, causing the duplicates to show up.

The build process is subtly different here between what happens today and what Cargo is expecting, but I think you can get around it by adding a dev dep on the test crate.

[cl91]

@alexcrichton Right that works! I've hit rust-lang/cargo#1779 along the way, so I have to rebuild Cargo, but I eventually got all the unit tests working for all the crates.

I'm now working on running the compiler test suites, which is the next major function of the old Makefile-based build system.