ARM builders

[yuyichao]

As @tkelman requested, here's a list of implemented/planned/proposed changes to how arm builders works.

1. I've finally got one of my AArch64 box to be stable enough for remote access and hosting automated jobs. I've created 2 LXC on the machine with Centos 7 (aarch64) and Debian 7 (armhf). Unless someone else (julia computing) acquires some other AArch64 servers, my plan was to use these as slaves for the buildbot since the server utilization is otherwise pretty low.
2. The two distros mentioned above are the ones with the oldest glibc I can possibly find. The Centos 7 has glibc 2.17, which I think is the first or the second version that supports aarch64. The Debian 7 has glibc 2.13, which isn't the first one that supports eabi (and possibly not the first one supporting eabihf) but is pretty close and I can't find anything older. (The oldest version of centos, fedora, ubuntu, opensuse that supports armhf all comes with a newer glibc).
3. The major advantage of the new server is performance. It should be 5-10x faster than the the old one and with more cores, memory and disk space.
4. IIUC the original plan was to integrate them in the newer version of buildbot (and I believe this is still the plan for the aarch64 one). However, the old arm buildbot somehow breaks recently and the gcc 6.2 doesn't compile there so the arm builder is already switched to the new server, which is clean and doesn't have the issue.
5. Before we can have a working nightly out from the new builder using gcc 6.2, the main blocking issue is Automatically detect ifunc support in the compiler JuliaLang/julia#18996, which I'll merge soon unless someone has a better solution. (I don't like it myself but can't find a better way...)
6. Since new builder is much faster and the unwinding issue is fixed, we can start running tests on arm. (We should also run aarch64 tests once the builder is online and all tests should pass on llvm 3.9). The known segfaulting tests there are disabled by Disable profile test on ARM by default JuliaLang/julia#19003 which I'll also merge soon.
7. The only disadvantage of the new builder that I can think of is that the uname -m is armv8l instead of armv6l or armv7l. This is only an issue when compiling any dependencies (noticeably clang and gcc) that picks up the target arch automatically from uname -m. Both of them should be overwritable. We can possibly figure this out from ARCH but I'm not sure what's the most robust way to do it.... In order to make sure the binary is compiled for the right arch, I think we can keep the old builder and run a few simple tests on it using the binary we compiled to make sure that the binary is good to use. From my brief testing just now it seems that the LLVM was probably not compiled with LTO and it is somehow using neon instructions.
8. We need to build for armv6 if we want to support rpi0 and rpi1. Do we want to add builder for this? (Also in any case the download link for the current binaries should say armv7 instead).

@staticfloat, @ViralBShah

[simonbyrne]

+1 to armv6, if possible

[yuyichao]

Current blocking issues for nightly update.

• Make sure we are compiling with LTO using gcc 6.2

  TL:DR. The relavant part of the command line to use for now is LLVM_CMAKE='-DCMAKE_CXX_FLAGS=-flto -DCMAKE_C_FLAGS=-flto -DCMAKE_EXE_LINKER_FLAGS=-flto -DCMAKE_SHARED_LINKER_FLAGS=-flto -DCMAKE_AR=$$(which gcc-ar) -DCMAKE_RANLIB=$$(which gcc-ranlib) -DCMAKE_NM=$$(which gcc-nm)' make LLVM_USE_CMAKE=1 LLVM_LTO=1 and I've tested it to make sure it works around all known issues I've seen.

  Reasons we needs such a complicated command line are,

  1. The LLVM version is currently not overwrite-able so we cannot use LLVM_ENABLE_LTO which is 3.9 only.
  2. Likely due to how the gcc 6.2 / binutils is configured we need to use the gcc wrapper for ar and ranlib. These aren't necessary when I tested with the stock gcc and binutils on arch linux.
  3. There's possibly llvm build system bugs on 3.8.1 which causes it to miss include path for certain files.
  4. We are not passing CFLAGS, CXXFLAGS and LDFLAGS to llvm with cmake build. This causes the LLVM_LTO flag from Add LLVM_LTO make flag to build LLVM with Link-Time Optimization JuliaLang/julia#19028 to be no-op, which both means we don't need that PR for now and that we need to specify the flags manually.... After this is fixed the command above should still work and can be simplified to LLVM_CMAKE='-DCMAKE_AR=$$(which gcc-ar) -DCMAKE_RANLIB=$$(which gcc-ranlib) -DCMAKE_NM=$$(which gcc-nm)' make LLVM_USE_CMAKE=1 LLVM_LTO=1 (and if we can figure out how to configure gcc correctly the env can be removed altogether).

• Merge Automatically detect ifunc support in the compiler JuliaLang/julia#18996

  This is the only change that blocks compilation so I'm waiting for other changes to be merged first so that we don't ship a nightly that does not work in certain configuration.

The slave is currently offline for me to debug the issue last night and it already has a correctly compiled LLVM cached (the one I compiled manually) so as long as the command above is implemented in the buildbot setup script we can merge JuliaLang/julia#18996 and ready to update the nightly.

I just breifly tried setting MARCH=armv6 locally and it's complaining because it picks up the -march flag from my llvm compilation........ I'll try to compile a LLVM and test it later. It would also be nice if we can have a armv6 box to test if the compiled binary works.

[yuyichao]

Update:

• There's a new issue with the cmake build of llvm 3.8.1 since it links libLTO.so which we don't ship (Thanks @vchuravy and @maleadt for the hint and the fix). This looks like an upstream bug and adding -DLLVM_LINK_LLVM_DYLIB=On works around it. The first commit in Bump LLVM version to 3.9 JuliaLang/julia#18863 addresses the same issue.

  The final command line I used was LLVM_CMAKE='-DLLVM_LINK_LLVM_DYLIB=On -DLLVM_HOST_TRIPLE=armv7l-unknown-linux-gnueabihf -DLLVM_DEFAULT_TARGET_TRIPLE=armv7l-unknown-linux-gnueabihf -DCMAKE_CXX_FLAGS=-flto -DCMAKE_C_FLAGS=-flto -DCMAKE_EXE_LINKER_FLAGS=-flto -DCMAKE_SHARED_LINKER_FLAGS=-flto -DCMAKE_AR=$$(which gcc-ar) -DCMAKE_RANLIB=$$(which gcc-ranlib) -DCMAKE_NM=$$(which gcc-nm)' make -j8 TAGGED_RELEASE_BANNER="Official http://julialang.org/ release" JULIA_CPU_TARGET=generic JULIA_CPU_CORES=4 JULIA_TEST_MAXRSS_MB=600 MARCH=armv7-a release LLVM_USE_CMAKE=1. The specified triples should also fix the uname issue mentioned above although the build seems to work without.

• With a working LLVM build cache and I've tested the resulting binary julianightlies.s3.amazonaws.com/bin/linux/arm/0.6/julia-0.6.0-5e99043224-linuxarm.tar.gz on scaleway so I've merged Automatically detect ifunc support in the compiler JuliaLang/julia#18996 which should make the next nightly build pass.

• We can use the cached LLVM build for now (so please don't nuke the arm builder for now). There are many issues with LLVM build noticed in this process so let's try to fix them properly in base rather than working around them on the buildbot. A list of those issues can be found Fix LLVM 3.8+ build JuliaLang/julia#19051.

• I've tested ARMv6 build and it seems to work afaict. I don't have an ARMv6 board and I don't remember the instructions that belongs to ARMv6 vs ARMv7 off the top of my head so I didn't check if all instructions are legal.I'm also using libraries from the ARMv7 system so we might still need to make sure dependencies other than LLVM (openblas/openlibm and maybe fftw in particular) works there. To move forward on this we'll probably need real hardware for testing.

Remaining TODO's (Assuming the new nightly is fine.....):

• Fix Fix LLVM 3.8+ build JuliaLang/julia#19051.
• Update the build settings for ARM. (With Fix LLVM 3.8+ build JuliaLang/julia#19051 fixed what we need to add should be LLVM_CMAKE='-DLLVM_HOST_TRIPLE=armv7l-unknown-linux-gnueabihf -DLLVM_DEFAULT_TARGET_TRIPLE=armv7l-unknown-linux-gnueabihf')
• Update LLVM version to 3.9. (There are bug fixes that are useful for ARM. Also relies on Fix LLVM 3.8+ build JuliaLang/julia#19051.)
• Enable full test run on the new ARM slave. I remember @tkelman told me there's some complicated setup for running nightly tests. Can we / how do we run tests using a nightly binary?
• Run a few simple tests on the old ARM slave (scaleway).
• Enable AArch64 slave (this is part of the new buildbot setup so we can probably close this issue without this one being implemented)

ARMv6 support should probably move to it's own issue.

[tkelman]

regarding adding tests, in the current arrangement if tests fail on any of the test builders then no binaries get built at all. we can't use that setup with arm, we're not blocking x86 binaries on arm bugs. so running tests should also wait until the buildbot setup is refactored.

[yuyichao]

The need for gcc trapper of ar, ranlib, nm is fixed. I did recompile binutils and gcc but I think what actually fixes it is to symlink the lto plugin into the binutils plugin directory (Ref https://github.com/archlinuxarm/PKGBUILDs/blob/527636ea075312c64aff6dad8ddc2b33933c8d62/core/gcc/PKGBUILD#L180-L183). The test build passed on the buildbot so nuking the buildbot should be fine now.

It'll still be nice to set the correct host triple, (which might actually be a regression in LLVM's cmake system).

[tkelman]

Cool, good catch. Is the ARM builder being managed by the same set of ansible scripts, should we record the need to do that somewhere so it's easy to re-provision if we ever need to?

[yuyichao]

@staticfloat for that. I think there's automatic script to do this and maybe he sent me a link to the repo (maybe https://github.com/staticfloat/julia-ansible-scripts ?).

For the record the command line I used are

• binutils

  ./configure --prefix=/home/debian/local --enable-lto --enable-plugins --disable-werror --host=armv7l-unknown-linux-gnueabihf --build=armv7l-unknown-linux-gnueabihf --enable-threads --enable-shared --with-pic --enable-ld=default --enable-gold --enable-deterministic-archives

  The --enable-lto --enable-plugins --host=armv7l-unknown-linux-gnueabihf --build=armv7l-unknown-linux-gnueabihf is likely the only important (arm/lto specific) part.

• gcc

  ../gcc_source/gcc-6.2.0/configure --prefix=/home/debian/local --with-gas=/home/debian/local/bin/gas --enable-host-shared --enable-threads=posix --enable-languages=c,c++,fortran,lto --disable-multilib --host=armv7l-unknown-linux-gnueabihf --build=armv7l-unknown-linux-gnueabihf --with-arch=armv7-a --with-float=hard --with-fpu=vfpv3-d16 --enable-lto --enable-plugin --disable-werror

  and the important (arm/lto specific) part is likely --enable-languages=c,c++,fortran,lto --disable-multilib --host=armv7l-unknown-linux-gnueabihf --build=armv7l-unknown-linux-gnueabihf --with-arch=armv7-a --with-float=hard --with-fpu=vfpv3-d16 --enable-lto --enable-plugin

I also needs to clear my CFLAGS since it triggers some Werror in gcc's dependencies that can't be turned off easily.

[yuyichao]

And with the above flags gcc installs the lto plugins to libexec by default and ln -s ~/local/libexec/gcc/armv7l-unknown-linux-gnueabihf/6.2.0/liblto_plugin.so ~/local/lib/bfd-plugins puts it to where binutils expects.

[staticfloat]

Confirmed that GCC 7 eliminates the need for us to do any LTO or AR/RANLIB overriding

[staticfloat]

Closed by #111