Add bin/build-ldc-runtime.sh helper script

[kinke]

Update:

This new script will automatically download and extract the full LDC source archive of the release it was packaged with and then build druntime/Phobos stand-alone (taking a few minutes) using

• user-supplied CMake command-line args,
• the LDC compiler in the same directory as the script for the D parts, and
• the CC C compiler for the C/ASM parts and linking the shared libs (on POSIX at least).

So we can build on top of that to let the user build the runtime libs with LTO support or for cross-compilation targets any time he/she wants to.

[kinke]

It'd be nice if someone on OSX could give this a try:

1. Download full LDC source from here.
2. Build LDC.
3. Tweak the download URL in bin/build_runtime.sh to the above source package.
4. Change to an arbitrary directory and run .../bin/build_runtime.sh -DD_FLAGS=-w;-flto=(full|thin) -DLD_FLAGS=-Wl,-lto_library,.../libLTO.dylib.

This should build static+shared druntime & Phobos libs with LTO support into ./build-runtime.tmp/lib. Starting with the next (beta) release, users would only need to execute the last step (and copy/replace the libs).

[kinke]

This fixes #2196 for me on Linux x64. Executing .../bin/build_runtime.sh -DMULTILIB=ON successfully generates 8 druntime/Phobos variants (debug/release, static/shared, 32/64 bits).

[JohanEngelen]

(for LTO, I think it'd be best to have LDC search for phobos/druntime with -LTO postfix when -flto is passed, instead of overwriting the non-LTO phobos/druntime)

[joakim-noah]

Interesting approach, I was thinking of using dub, since we now package it alongside ldc anyway. This would mean no dependency on CMake or Make, only a C cross-compiler for the few C/asm files the stdlib uses, but presumably would require writing a whole new dub.sdl that builds the stdlib. I have not looked at or used dub enough to know if that's a good idea, thoughts?

Another possibility is that we could bundle in reggae and the Phobos reggaefile that Atila already wrote for us, along with one we write for druntime. I'm not sold on D syntax for build scripts, but it's certainly better than CMake.

[kinke]

I think the CMake (a pretty old version should suffice, only LLVM's CMake scripts require a pretty recent version) and make requirements aren't a big hurdle for interested users.

Moving from CMake to a dub or reggae script for the runtime libs may be worth the trouble if the resulting script ends up clearly more compact and more readable while supporting the same platforms. Looking at Atila's 500-lines reggae script for Phobos alone, I doubt that goal is easily achievable, given the numerous special cases in our current 830-lines CMake script (BUILD_BC_LIBS, profile-rt library, building static+shared libs in a single pass with shared Phobos object files, creating the final ldc2.conf config files, setting up the CTest tests...).

[joakim-noah]

I think the CMake (a pretty old version should suffice, only LLVM's CMake scripts require a pretty recent version) and make requirements aren't a big hurdle for interested users.

You're probably right for many, as setting up a C cross-compilation toolchain usually isn't easy, and still required if they want to do anything with our cross-compiler. But at that point, we could just tell them to git clone the ldc repo too, it's only one more step after all. ;) The beauty of this PR's approach is that it requires almost no change to our existing build process, but that's also its flaw: it has all the same dependencies.

Moving from CMake to a dub or reggae script for the runtime libs may be worth the trouble if the resulting script ends up clearly more compact and more readable while supporting the same platforms. Looking at Atila's 500-lines reggae script for Phobos alone, I doubt that goal is easily achievable, given the numerous special cases in our current 830-lines CMake script (BUILD_BC_LIBS, profile-rt library, building static+shared libs in a single pass with shared Phobos object files, creating the final ldc2.conf config files, setting up the CTest tests...).

Do we really want to support all that for the compiler release though? The reason Atila's script is so large is that it translates all build-related steps from Phobos's posix.mak, and ends up maybe 15-20% shorter. We could pare it down to just building druntime/phobos, along with LTO and some cross-compilation targets, and get it or some dub build files much smaller.

I've never written a dub build file, any one have an opinion on it?

[kinke]

But at that point, we could just tell them to git clone the ldc repo too, it's only one more step after all. ;) The beauty of this PR's approach is that it requires almost no change to our existing build process, but that's also its flaw: it has all the same dependencies.

Well, there's no git dependency for example. ;) - It should be really only CMake ≥ 2.8.9 on top of an anyway-required C toolchain incl. make [as well as bash, wget and tar]. I.e., no need for git, LLVM libs and headers, Python and recent CMake + gcc ≥ 4.9 (for LLVM) as required for a full LDC build.

[kinke]

Do we really want to support all that for the compiler release though?

You mean keeping the existing CMake script for the stdlibs produced as part of the full LDC build and adding a new lightweight dub script for standalone stdlib building? If it's very compact and doesn't need a lot of maintenance, that may be an option, but it'll have to compete against this proposed 30-lines bash script with 0 maintenance cost and full configurability. ;)

[joakim-noah]

Yes, I'm talking about having a separate dub script for quickly building druntime and phobos from the release, with a few options for cross-compiling, LTO, and the like. Quickly looking at some dub scripts, they seem almost impossibly small, might be a good fit, particularly if we can get rid of the CMake/make dependencies too. We'd have to actually include all druntime and phobos source in the ldc binary release though, some of which we don't ship now.

I'll look at it after I finally get my cross-compilation PR up for the runtime's CMake file, the last remaining bit from my Android fork of ldc.

[kinke]

Updated: don't install the bin/build_runtime.sh script for CMAKE_INSTALL_PREFIX = "/usr" and don't depend on GNU readlink (apparently not available by default on OSX); add commit msg.

[kinke]

I'll look at it after I finally get my cross-compilation PR up for the runtime's CMake file

That was exactly one use case I had in mind for this script. Wouldn't something like CC=cross-gcc .../build_runtime.sh -DBUILD_SHARED_LIBS=OFF -DD_FLAGS="-w;-mtriple=..." fit the bill, at least on Linux hosts (due to CMake host OS checks instead of target OS checks)?

[joakim-noah]

Wouldn't something like CC=cross-gcc .../build_runtime.sh -DBUILD_SHARED_LIBS=OFF -DD_FLAGS="-w;-mtriple=..." fit the bill, at least on Linux hosts (due to CMake host OS checks instead of target OS checks)?

Can you pass RT_CFLAGS that way? The main thing I want to enable with my PR is cross-compiling the test runners easily, which will require linker flags too.

[kinke]

Sure - you can specify all CMake variables in runtime/CMakeLists.txt this way, incl. RT_CFLAGS for the extra C flags and LD_FLAGS for the linker flags.

[joakim-noah]

OK, I was thinking of including a list of RT_CFLAGS and RT_LDFLAGS for cross-compilation targets we support, that could be enabled with a single platform flag and which could always be over-ridden by advanced users who want to pass their own. I didn't know about LD_FLAGS, as it appears to only be used in multilib and a few other places, and doesn't look to be currently passed when linking the test runners.

[kinke]

Afaict, LD_FLAGS are currently only used for the shared stdlibs. I had a very quick look, and apparently the testrunners are linked via LDC, so as a hack, the testrunner linker flags could be passed to LDC via D_FLAGS. But using LDC for cross-compiling and -linking the testrunners doesn't seem to be the way to go; I think we should compile test_runner.d separately and let CMake handle linking (via CC and LD_FLAGS), just like we do for the stdlibs.

And at some point, we should replace the host OS checks via a TARGET_OS CMake variable, defaulting to the host OS.

I was thinking of including a list of RT_CFLAGS and RT_LDFLAGS for cross-compilation targets we support

Sounds good. I never intended to release this script alone; the plan is to build on top of it via some handy wrapper scripts for LTO and cross-compilations and some way of installing the freshly built libs incl. ldc2.conf tweaking.

[kinke]

Any objections? I'll probably rename the script to build_ldc_runtime.sh to avoid name clashes; better name proposals are welcome.

[JohanEngelen]

I would like it if I could run this also from my dev setup (i.e. from a git checkout of master). Perhaps add a way to point it to a source tree and bypass the download?

[JohanEngelen]

no objections, other than renaming the build.sh.in file to whatever you name the cmake-configure-output script file.

[joakim-noah]

No objection, seems fine.

[kinke]

Perhaps add a way to point it to a source tree and bypass the download?

Yep, makes sense.

[kinke]

Alright, script reworked to take into account optional env variables LDC_SRC_DIR and BUILD_DIR. It's also friendlier to incremental builds by only warning about a pre-existing build dir and re-using the downloaded LDC source tree in there. Very handy when porting the libs to a new platform and ironing out the issues.