Null pointer exception in NonconfigurableAttributeMapper.get fetching 'testonly' attribute causing "Unrecoverable error while evaluating node"

[ivucica]

Description of the problem / feature request / question:

Bazel is crashing in a confusing way that does not provide actual feedback on what went wrong. It seems to be checking testonly attribute on one of the rules, but which one, I can't tell.

If possible, provide a minimal example to reproduce the problem:

git clone https://github.com/ivucica/grpc
cd grpc
git checkout bazelcrash # a.k.a. 2842a4c, or ivucica_workspace a.k.a. 9066117
bazel build :grpc

Environment info

• Operating System:
  Debian testing

• Bazel version (output of bazel info release):
  release 0.4.0

• If bazel info release returns "development version" or "(@non-git)", please tell us what source tree you compiled Bazel from; git commit hash is appreciated (git rev-parse HEAD):
  n/a

Stacktrace

ivucica@ivucica:~/priv/grpc$ bazel build :grpc
Unhandled exception thrown during build; message: Unrecoverable error while evaluating node 'CONFIGURED_TARGET://:grpc 68bff2929a60168b2fc19ce4ff23de5f (1035322455 421361641)' (requested by nodes )
INFO: Elapsed time: 3.197s
java.lang.RuntimeException: Unrecoverable error while evaluating node 'CONFIGURED_TARGET://:grpc 68bff2929a60168b2fc19ce4ff23de5f (1035322455 421361641)' (requested by nodes )
	at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:429)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:496)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
	at com.google.devtools.build.lib.packages.NonconfigurableAttributeMapper.get(NonconfigurableAttributeMapper.java:45)
	at com.google.devtools.build.lib.bazel.rules.BazelRuleClassProvider$BazelPrerequisiteValidator.isTestOnlyRule(BazelRuleClassProvider.java:252)
	at com.google.devtools.build.lib.bazel.rules.BazelRuleClassProvider$BazelPrerequisiteValidator.validateDirectPrerequisiteForTestOnly(BazelRuleClassProvider.java:239)
	at com.google.devtools.build.lib.bazel.rules.BazelRuleClassProvider$BazelPrerequisiteValidator.validate(BazelRuleClassProvider.java:182)
	at com.google.devtools.build.lib.analysis.RuleContext$Builder.validateDirectPrerequisite(RuleContext.java:1924)
	at com.google.devtools.build.lib.analysis.RuleContext$Builder.createTargetMap(RuleContext.java:1597)
	at com.google.devtools.build.lib.analysis.RuleContext$Builder.build(RuleContext.java:1436)
	at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createRule(ConfiguredTargetFactory.java:240)
	at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createConfiguredTarget(ConfiguredTargetFactory.java:171)
	at com.google.devtools.build.lib.skyframe.SkyframeBuildView.createConfiguredTarget(SkyframeBuildView.java:488)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.createConfiguredTarget(ConfiguredTargetFunction.java:984)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(ConfiguredTargetFunction.java:238)
	at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:370)
	... 4 more
java.lang.RuntimeException: Unrecoverable error while evaluating node 'CONFIGURED_TARGET://:grpc 68bff2929a60168b2fc19ce4ff23de5f (1035322455 421361641)' (requested by nodes )
	at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:429)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:496)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
	at com.google.devtools.build.lib.packages.NonconfigurableAttributeMapper.get(NonconfigurableAttributeMapper.java:45)
	at com.google.devtools.build.lib.bazel.rules.BazelRuleClassProvider$BazelPrerequisiteValidator.isTestOnlyRule(BazelRuleClassProvider.java:252)
	at com.google.devtools.build.lib.bazel.rules.BazelRuleClassProvider$BazelPrerequisiteValidator.validateDirectPrerequisiteForTestOnly(BazelRuleClassProvider.java:239)
	at com.google.devtools.build.lib.bazel.rules.BazelRuleClassProvider$BazelPrerequisiteValidator.validate(BazelRuleClassProvider.java:182)
	at com.google.devtools.build.lib.analysis.RuleContext$Builder.validateDirectPrerequisite(RuleContext.java:1924)
	at com.google.devtools.build.lib.analysis.RuleContext$Builder.createTargetMap(RuleContext.java:1597)
	at com.google.devtools.build.lib.analysis.RuleContext$Builder.build(RuleContext.java:1436)
	at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createRule(ConfiguredTargetFactory.java:240)
	at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createConfiguredTarget(ConfiguredTargetFactory.java:171)
	at com.google.devtools.build.lib.skyframe.SkyframeBuildView.createConfiguredTarget(SkyframeBuildView.java:488)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.createConfiguredTarget(ConfiguredTargetFunction.java:984)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(ConfiguredTargetFunction.java:238)
	at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:370)
	... 4 more

[laszlocsomor]

Thanks for the super easy repro case. I can see this happening too. Investigating.

[laszlocsomor]

The problem is triggered by //:grpc depending on //external:libssl. The latter is a http_archive rule and I don't know what's the expected behavior if a cc_library depends on such a rule, perhaps it will depend on @libssl//:*.

@kchodorow , @gregestren : Should this kind of dependency be allowed? What are the expected semantics?

[applmak]

I ran into this and employed a workaround via 'bind':

new_*(
  name = "$THING_repo",
  ...
)

bind(
  name = "$THING",
  actual = "@$THING_repo//:$THING"
)

e.g.:

git_repository(
  name = "nanopb_repo",
  remote = "https://github.com/nanopb/nanopb.git",
  commit = "60d8ba229b58f07fdefc31e2756402cc74cede5f"
)

bind(
  name = "nanopb",
  actual = "@nanopb_repo//:nanopb"
)

[laszlocsomor]

@applmak : Thanks for the workaround!

[ivucica]

I think this is probably a non-issue, aside from the error being incomprehensible.

[llhe]

@laszlocsomor How to identify it is caused by '//external:libssl'? I have the same problem when include grpc into tensorflow build. But I can figure out which one is the cause. Does bazel support verbose mode?

[laszlocsomor]

@llhe : Are you also seeing an uncaught exception or is it some other error?
I'd first try to comment out the "//external:libssl" dependency from grpc's BUILD file and see if the project builds normally (or fails with a nicer error message, as opposed to just throwing).
You can also build with the --verbose_failures flag, but that's only effective if the build does not throw.

It's always a bug in Bazel if there's an uncaught exception. Let me ping the assignees.

[llhe]

@laszlocsomor Yes, I encountered this issue. I use the most recent grpc code, and it's caused by "protobuf" dependency. I identified this by actually modified the bazel source code to to print the info I needed.

Yes, bazel should have a fix to make a not null check here, I can provide this simple patch that I use.

[gregestren]

I'm not reproducing now because of an outdated zlib dependency. What's the current status of the repro?

[qzmfranklin]

Hey guys, I seem to have run into this same issue recently.

I had the following snippet in the WORKSPACE file:

git_repository(
    name = 'grpc',
    remote = 'https://github.com/grpc/grpc',
    tag = 'v1.4.6',
)

bind(
    name = 'grpc_cpp_plugin',
    actual = '@grpc//:grpc_cpp_plugin',
)

Then I issued the following command:

bazel build //external:grpc_cpp_plugin

and got the following error messages:

bazel build //external:grpc_cpp_plugin
Unhandled exception thrown during build; message: Unrecoverable error while evaluating node 'CONFIGURED_TARGET:@grpc//:grpc++_config_proto a6d8de05646caacaeb56a773d79fc7a5 (1987867078 291368319)' (requested by nodes 'CONFIGURED_TARGET:@grpc//:grpc_plugin_support a6d8de05646caacaeb56a773d79fc7a5 (1005836117 291368319)')
INFO: Elapsed time: 49.898s
FAILED: Build did NOT complete successfully (6 packages loaded)
java.lang.RuntimeException: Unrecoverable error while evaluating node 'CONFIGURED_TARGET:@grpc//:grpc++_config_proto a6d8de05646caacaeb56a773d79fc7a5 (1987867078 291368319)' (requested by nodes 'CONFIGURED_TARGET:@grpc//:grpc_plugin_support a6d8de05646caacaeb56a773d79fc7a5 (1005836117 291368319)')
	at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:476)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:352)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: No such attribute testonly in git_repository rule //external:protobuf
	at com.google.devtools.build.lib.packages.AbstractAttributeMapper.getIndexWithTypeCheck(AbstractAttributeMapper.java:229)
	at com.google.devtools.build.lib.packages.AbstractAttributeMapper.get(AbstractAttributeMapper.java:57)
	at com.google.devtools.build.lib.packages.NonconfigurableAttributeMapper.get(NonconfigurableAttributeMapper.java:45)
	at com.google.devtools.build.lib.bazel.rules.BazelPrerequisiteValidator.isTestOnlyRule(BazelPrerequisiteValidator.java:121)
	at com.google.devtools.build.lib.bazel.rules.BazelPrerequisiteValidator.validateDirectPrerequisiteForTestOnly(BazelPrerequisiteValidator.java:104)
	at com.google.devtools.build.lib.bazel.rules.BazelPrerequisiteValidator.validate(BazelPrerequisiteValidator.java:36)
	at com.google.devtools.build.lib.analysis.RuleContext$Builder.validateDirectPrerequisite(RuleContext.java:2123)
	at com.google.devtools.build.lib.analysis.RuleContext$Builder.createTargetMap(RuleContext.java:1787)
	at com.google.devtools.build.lib.analysis.RuleContext$Builder.build(RuleContext.java:1614)
	at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createRule(ConfiguredTargetFactory.java:295)
	at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createConfiguredTarget(ConfiguredTargetFactory.java:220)
	at com.google.devtools.build.lib.skyframe.SkyframeBuildView.createConfiguredTarget(SkyframeBuildView.java:489)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.createConfiguredTarget(ConfiguredTargetFunction.java:1125)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(ConfiguredTargetFunction.java:253)
	at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:401)
	... 4 more
java.lang.RuntimeException: Unrecoverable error while evaluating node 'CONFIGURED_TARGET:@grpc//:grpc++_config_proto a6d8de05646caacaeb56a773d79fc7a5 (1987867078 291368319)' (requested by nodes 'CONFIGURED_TARGET:@grpc//:grpc_plugin_support a6d8de05646caacaeb56a773d79fc7a5 (1005836117 291368319)')
	at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:476)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:352)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: No such attribute testonly in git_repository rule //external:protobuf
	at com.google.devtools.build.lib.packages.AbstractAttributeMapper.getIndexWithTypeCheck(AbstractAttributeMapper.java:229)
	at com.google.devtools.build.lib.packages.AbstractAttributeMapper.get(AbstractAttributeMapper.java:57)
	at com.google.devtools.build.lib.packages.NonconfigurableAttributeMapper.get(NonconfigurableAttributeMapper.java:45)
	at com.google.devtools.build.lib.bazel.rules.BazelPrerequisiteValidator.isTestOnlyRule(BazelPrerequisiteValidator.java:121)
	at com.google.devtools.build.lib.bazel.rules.BazelPrerequisiteValidator.validateDirectPrerequisiteForTestOnly(BazelPrerequisiteValidator.java:104)
	at com.google.devtools.build.lib.bazel.rules.BazelPrerequisiteValidator.validate(BazelPrerequisiteValidator.java:36)
	at com.google.devtools.build.lib.analysis.RuleContext$Builder.validateDirectPrerequisite(RuleContext.java:2123)
	at com.google.devtools.build.lib.analysis.RuleContext$Builder.createTargetMap(RuleContext.java:1787)
	at com.google.devtools.build.lib.analysis.RuleContext$Builder.build(RuleContext.java:1614)
	at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createRule(ConfiguredTargetFactory.java:295)
	at com.google.devtools.build.lib.analysis.ConfiguredTargetFactory.createConfiguredTarget(ConfiguredTargetFactory.java:220)
	at com.google.devtools.build.lib.skyframe.SkyframeBuildView.createConfiguredTarget(SkyframeBuildView.java:489)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.createConfiguredTarget(ConfiguredTargetFunction.java:1125)
	at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(ConfiguredTargetFunction.java:253)
	at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:401)
	... 4 more

I use bazel release 0.5.3 on Ubuntu 17.04 amd64.

Could anyone confirm that this issue has been patched and/or can be fixed?

[gregestren]

This looks like a pretty straightforward bug, fixable by simply modifying:

bazel/src/main/java/com/google/devtools/build/lib/bazel/rules/BazelPrerequisiteValidator.java

Line 130 in 91fb38e

&& (NonconfigurableAttributeMapper.of((Rule) target)).get("testonly", Type.BOOLEAN);

to also add an attribute existence check:

bazel/src/main/java/com/google/devtools/build/lib/packages/AbstractAttributeMapper.java

Line 250 in b13457b

public boolean has(String attrName) {

I'd do this myself, but I'll be out on vacation imminently so I may not be able to get around to it until September. If anyone wants to try a patch they're welcome. Otherwise ping me post-vacation and I'll fix this up.

[zlalanne]

@gregestren I think I'm running into this, did you ever end up making a patch?

[gregestren]

I didn't. But as luck would have it this also popped up in #3676 and was fixed at head 2.5 weeks ago. So this should be in the next Bazel release.

[gregestren]

Duplicate of #3676