bazel --keep_going exits successfully when encountering starlark errors in package loading

[schroederc]

Description of the problem / feature request:

Bazel will return successfully from a build or test run that includes package load errors when given --keep_going.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

$ git clone https://github.com/google/kythe.git
$ cd kythe
$ git checkout c3919636ac995d78ceeef09b1e5b84d287b097c8
$ bazel-0.20.0 build --keep_going --nobuild //kythe/cxx/indexer/cxx/...
ERROR: kythe/tools/build_rules/verifier_test/cc_indexer_test.bzl:153:43: name 'ProtoInfo' is not defined
ERROR: error loading package 'kythe/cxx/indexer/cxx/testdata/proto': in kythe/kythe/cxx/indexer/cxx/testdata/proto/cc_proto_verifier_test.bzl: Extension 'tools/build_rules/verifier_test/cc_indexer_test.bzl' has errors
ERROR: error loading package 'kythe/cxx/indexer/cxx/testdata': Extension 'tools/build_rules/verifier_test/cc_indexer_test.bzl' has errors
INFO: Analysed 23 targets (0 packages loaded, 0 targets configured).
INFO: Found 23 targets...
INFO: Elapsed time: 0.191s
INFO: 0 processes.
INFO: Build completed successfully, 0 total actions

# Bazel also returns a 0 exit code; happens without --nobuild as well.

What operating system are you running Bazel on?

Linux

What's the output of bazel info release?

release 0.20.0

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

https://github.com/kythe/kythe.git
c3919636ac995d78ceeef09b1e5b84d287b097c8
c3919636ac995d78ceeef09b1e5b84d287b097c8

[aiuto]

It sounds like the right solution is that blaze should return non-zero if there were errors, regardless of them being package loading related or not.

[alexeagle]

We pushed a bad commit to rules_typescript yesterday because of this issue.
/subscribe

[haxorz]

b8ac792 probably fixed this. i believe 0.23.0 was the first bazel release that contained that commit. can anyone complaining about this issue please try again with that bazel release (or a newer one)?

i tried with the original repro instructions but was getting errors due to incompatibilities between bazel 0.23.0 and kythe @ c3919636ac995d78ceeef09b1e5b84d287b097c8.

[schroederc]

This is still an issue in 0.24.0. You can repro by attempting to reference any undefined variable in a loaded .bzl file. That will cause the BUILD package load to fail.

[philwo]

@haxorz It seems like @sergiocampama encountered this in rules_apple again. I'm copying his internal bug report (b/133148805) here:

Please see https://buildkite.com/bazel/rules-apple-darwin/builds/883#_ and open the logs for "Latest Bazel on Apple". This job is marked as successful (green check), but in reality it failed to run some tests because of a Starlark issue. We presume this is because of the --keep_going flag.

Either Bazel should return a non successful code when there is such a failure, or BuildKite should detect that there was an error in the build and should mark the job as such. Possibly both.

[sergiocampama]

I wonder how many projects in BuildKite may be silently broken (rules_apple definitely is)

[haxorz]

i or someone else will fix this (, probably internally).

assigning to myself for tracking purposes.