Bazel@HEAD broke toolchains

[hlopko]

Hi all,

tonight downstream pipeline was red because of toolchain_identifier error: https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/498#d982b428-5036-41e2-9edc-1107b286d850

This might have been caused by bazelbuild/bazel@c3fb1db or by (now rolledback) bazelbuild/bazel@3aedb2f (leaving for the plane in 3 minutes, cannot investigate more).

Can you help me understand what caused the failure, and whether you think the new behavior should be guarded with an incompatible flag (https://docs.bazel.build/versions/master/skylark/backward-compatibility.html#disallow-using-crosstool-to-select-the-cc_toolchain-label).

[xingao267]

It's probably due to bazelbuild/bazel@c3fb1db. The error message is "ERROR: Toolchain identifier 'local' was not found, valid identifiers are [stub_armeabi-v7a, linux_gnu_x86, msys_x64_mingw, msvc_x64]".

The tests that are failing are doing a simple cc autoconfig using Bazel@HEAD in a container (doing bazel build --all_incompatible_changes @local_config_cc//... using bazel@HEAD in a container).

I'm trying to understand why "local" is no longer a valid toolchain identifier.

[hlopko]

Cc @meteorcloudy. Ha I finally know what's going on. I'm rolling back the bazelbuild/bazel@c3fb1db, but the fix has to be made in bazel-toolchains. The problem is that you're building @local_config_cc//..., but with a different CROSSTOOL (because you specify your own toolchain using --crosstool_top). That results in @local_config_cc//tools/cpp:cc-compiler-k8 to be used with a different CROSSTOOL, and failing. I consider building cc_toolchain with a different CROSSTOOL to be unsupported use case and therefore outside of backwards incompatible policy. And FYI this would just work as expected if local_config_cc used (cc_toolchain_suite|cc_toolchain).proto attribute, or when we'll migrate to crosstool in skylark (bazelbuild/bazel#5380).

Today it works just because toolchain selection is super benevolent and will accept anything that even remotely looks like could be used. I'm fixing that. I'm sure that's not what you intended to test.

I guess the fix is not to set --crosstool_top when testing local_config_cc.

[hlopko]

I discussed this with colleagues and decided to temporarily remove bazel-toolchains from Bazel with downstream projects pipeline. The problem is that to roll out my change I'd have to create incompatible environment variable, and duplicate all the affected files, and there's no infrastructure to handle incompatible environment variables in place. And since we consider this behavior a bug, we'll not follow the policy (also because the policy is not yet set in stone :)
Thanks for understanding.

[nlopezgi]

Hi Marcel,
Sorry for these issue. I should be able to fix these tests by not using the --all_incompatible_changes flag for these builds. Will follow up soon with a PR.

[xingao267]

@mhlopko we tried removed the --all_incompatible_changes flag in #203, but it looks like the tests are still failing due to the same reason. https://source.cloud.google.com/results/invocations/169760d3-f703-4f0a-9bfd-d8f853042c99/targets/%2F%2Ftest%2Fconfigs:debian-jessie-bazel-head-autoconfig_test/log

I'm going to temporarily disable the two failing tests in the presubmit.

[hlopko]

The real issue is that you build @local_config_cc//..., that builds all cc_toolchain_suite and cc_toolchain targets inside, but without using their corresponding CROSSTOOL (that is specified by --crosstool_top, and I assume you specify it to point to your own toolchain suite). Am I making any sense?

[xingao267]

I'm a little confused. In this test, we don't specify --crosstool_top. We just run bazel build @local_config_cc//... in a container to let Bazel autoconfig itself, and then we extract the generated CROSSTOOL and BUILD files. These configs will later be used for remote execution together with that same container.

[xingao267]

Or maybe something is changed for how we should obtain those cc toolchain files?

[hlopko]

And there's no --crosstool_top anywhere (e.g. bazelrc)? Then I misunderstood the issue. But I do observe cc_toolchain/CROSSTOOL mismatch. Is it possible that you patch or manipulate with the @local_config_cc//tools/cpp:CROSSTOOL somehow?

[nlopezgi]

Thanks for all the comments Marcel.
I was able to debug further, and it seems this error is occurring because we have some env variables set in our containers. Specifically, if I remove the following env variable, the tests pass:
export CC_TOOLCHAIN_NAME="linux_gnu_x86"
Is it wrong to set this env variable now (it used to work?)?
How can we set this env variable to some value so that our toolchain identifier will not come out as "local" (which does not make much sense for rbe) in here (https://github.com/bazelbuild/bazel-toolchains/blob/master/configs/ubuntu16_04_clang/1.1/bazel_0.17.1/default/CROSSTOOL#L89) (but in a way that avoids this error).

[hlopko]

Thanks Nick, that was valuable debugging! This is exactly the problem. We use CC_TOOLCHAIN_NAME to set the toolchain_identifier in the CROSSTOOL, but not in the BUILD file. I'll fix it today. I apologise for the overhead you had to have for our bug.