cc_toolchain_suite does not contain a toolchain for cpu 'k8'

[lane-s]

I'm getting the following error when attempting to build envoy with rpi_bazel:

ERROR: ~/.cache/bazel/_bazel_my_usr_name/05c2e2981e4691f39b198a95ef8b9a7c/external/rpi_bazel/tools/cc_toolchain/BUILD:5:1: in cc_toolchain_suite rule @rpi_bazel//tools/cc_toolchain:toolchain: cc_toolchain_suite '@rpi_bazel//tools/cc_toolchain:toolchain' does not contain a toolchain for cpu 'k8'
ERROR: Analysis of target '//source/exe:envoy-static' failed; build aborted: Analysis of target '@rpi_bazel//tools/cc_toolchain:toolchain' failed; build aborted

I verified that I could build envoy with bazel normally on my machine before attempting to cross compile for rpi. I'm guessing the error has something to do with the toolchain definitions in the BUILD or CROSSTOOL file in tools/cc_toolchain, but the error doesn't make much sense to me since in the CROSSTOOL file the default toolchain for k8 clearly points to the clang x86_64 toolchain. I will keep trying to figure it out, but any help would be appreciated.

Some system info in case that's relevant:
Operating System: Linux Mint 19 Kernel: Linux 4.15.0-34-generic Architecture: x86-64

Bazel info:

Build label: 0.23.2
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Mar 11 16:47:03 2019 (1552322823)
Build timestamp: 1552322823
Build timestamp as int: 1552322823

[jpieper]

Maybe try specifying the compiler too?

bazel test --cpu=k8 --compiler=clang-6.0 //whatever:target

See https://github.com/mjbots/rpi_bazel/blob/master/.bazelrc#L1

[lane-s]

Thanks! That allowed it to start the build process at least, but I'm stumped again: now clang fails to compile a test program:

  The C compiler

    "/usr/bin/clang-6.0"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /tmp/tmp.mVPZnR5jL8/CMakeFiles/CMakeTmp
    
    Run Build Command:"/usr/bin/ninja" "cmTC_54f7a"
    [1/2] Building C object CMakeFiles/cmTC_54f7a.dir/testCCompiler.c.o
    [2/2] Linking C executable cmTC_54f7a
    FAILED: cmTC_54f7a 
    : && /usr/bin/clang-6.0 --target=armv7-linux-gnueabihf -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -DMB_LEN_MAX=16 -fstack-protector -Wall -B/usr/bin -B/usr/bin -fcolor-diagnostics -ffreestanding -fno-omit-frame-pointer -isysroot=/home/my_user_name/.cache/bazel/_bazel_my_user_name/05c2e2981e4691f39b198a95ef8b9a7c/sandbox/linux-sandbox/181/execroot/envoy/external/raspberry_pi/sysroot --sysroot=/home/melodylane/.cache/bazel/_bazel_my_user_name/05c2e2981e4691f39b198a95ef8b9a7c/sandbox/linux-sandbox/181/execroot/envoy/external/raspberry_pi/sysroot -g -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted"  -rdynamic CMakeFiles/cmTC_54f7a.dir/testCCompiler.c.o  -o cmTC_54f7a   && :
    /usr/bin/ld: unrecognised emulation mode: armelf_linux_eabi
    Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu i386linux elf_l1om elf_k1om i386pep i386pe

I can't seem to find any information about what determines the supported emulations for ld, but it's clear from running ld -V that armelf_linux_eabi is not supported currently. I'm not sure if this is system dependent or if there's something I can install to add support. I tried installing gcc-arm-none-eabi, but that had no effect on the supported emulations.

[jpieper]

First: rpi_bazel downloads and runs a custom clang-6 toolchain, you shouldn't be using your system one.
Second: Your problem there is that you are using the gnu linker, instead of the clang linker. rpi_bazel should use the correct linker if you build from within bazel.

[jpieper]

Apologies, I was thinking of a different repository w.r.t. the first point. rpi_bazel does use the system installed clang-6.0 (although I should probably switch it to use a vendored version).

The problem remains though that it looks like you were building with cmake/ninja, not bazel, and it was using the wrong linker.

[lane-s]

Hmm, so the error occurs after running bazel build --compiler=clang-6.0 --config=pi //target while bazel attempts to build the 'zlib' library. As you noted, it looks like this delegates to an external build script that uses CMake and ninja. I don't think that's something I have much control over, but maybe I could modify the build script to use the correct linker?

[jpieper]

Yep, that looks like envoy is not using bazel all throughout, but shells out to other build systems, which don't know about the bazel configured toolchain.

For that, yep, you'll have to get everything else cross compiling properly using either their native build system, or by porting their build system to bazel. I've ported many things to bazel, (including zlib) at https://github.com/mjbots/bazel_deps

https://github.com/mjbots/bazel_deps/tree/master/tools/workspace/zlib

[lane-s]

Ok, thanks for clearing that up for me. So if I were to successfully cross compile each of the dependencies that don't use bazel, then I'm assuming it would be fairly straightforward to configure the envoy build process to use my cross compiled binaries/libraries rather than trying to build them again using these scripts. I'm used to just writing application code, so this is all pretty foreign to me, but you've been a really big help so thanks again.

[jpieper]

It may be possible to configure envoy to use precompiled things, but is probably annoyingly difficult. I'm not familiar with envoy itself, but for some projects, the number of dependencies can be very large, and cross compiling for many things is a big project unto itself.

I spent several months on the things ported to bazel_deps, and similarly projects like yocto have invested lots of development effort in making it happen.

[lane-s]

Good to know. Apparently someone has built envoy for for ARM solely using CMake, so I will try to follow in their footsteps to see if I can figure that out.