Support zipped source files for cc_library and other languages

[ekuefler]

java_library allows .srcjars to be passed in srcs, which will be unzipped and all of their sources included in compilation. There is no equivalent for cc_library or other languages. This would be extremely useful to support genrules that create sets of source files that might not be fully defined at analysis time. See https://groups.google.com/forum/#!topic/bazel-discuss/O0MwYZLBaKU for more discussion.

It looks to me like the change would have to be made around CcCommon.java:254. The code to deal with source jars in Java is at AbstractLibraryBuilder.java:209

[ulfjack]

We are currently working on adding support for having rules that can generate a set of output files not all of which can be known at analysis time, and passing the entire set to a downstream action or alternatively running one action per output (and maybe other schemes as well).

[ulfjack]

Parts of the infrastructure are checked in, but it requires rule changes for each rule that wants to support it.

[ulfjack]

Unassigning myself to clarify that I'm not working on this right now.

[hlopko]

This is now implemented for tree artifacts (not only zip archives). Example on stack overflow: https://stackoverflow.com/questions/48417712/how-to-build-static-library-from-the-generated-source-files-using-bazel-build.

I'll close this issue now, please comment if I'm wrong for C++, or comment/create corresponding issues for other languages. Thanks!

[marcohu]

While this approach may work for C++, it does not for Java. Given a directory as input, java_library either complains that "a .java, .srcjar or .properties file is expected", or fails if the directory ends with the .srcjar extension (because a zip archive is assumed).

Are there any plans to provide a truly uniform approach for the genrules use case? Having to add a language specific extension does not work well for me.

I have a generator for multiple languages and would rather not have to adjust the output method depending on the language, because the output can vary depending on the input which would require an additional analysis up-front which might be difficult in Skylark.

The .srcjar approach works quite nicely with Java. Is the same not feasible with other languages?

[marcohu]

I just tried the described approach for C++. It fails if the directory contains header files. I'm using Bazel 0.12.0.

My use case looks like this:

BUILD

load("@rules_antlr//antlr:antlr4.bzl", "antlr4")
antlr4(
    name = "generated",
    srcs = ["src/main/antlr4/Hello.g4"],
    language = "Cpp",
)
cc_library(
    name = "Native",
    srcs = [":generated"],
)

It yields the following exception:

Unhandled exception thrown during build; message: Unrecoverable error while evaluating node 'ActionTemplateExpansionKey{actionLookupKey=//antlr4/Native:Native com.google.devtools.build.lib.skyframe.BuildConfigurationValue$Key@2795597c false (392459051), actionIndex=1}' (requested by nodes 'antlr4/Native/_pic_objs/Native/antlr4/Native/generated.cc //antlr4/Native:Native com.google.devtools.build.lib.skyframe.BuildConfigurationValue$Key@2795597c false (392459051)')
INFO: Elapsed time: 1.025s, Critical Path: 0.75s
FAILED: Build did NOT complete successfully
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionTemplateExpansionKey{actionLookupKey=//antlr4/Native:Native com.google.devtools.build.lib.skyframe.BuildConfigurationValue$Key@2795597c false (392459051), actionIndex=1}' (requested by nodes 'antlr4/Native/_pic_objs/Native/antlr4/Native/generated.cc //antlr4/Native:Native com.google.devtools.build.lib.skyframe.BuildConfigurationValue$Key@2795597c false (392459051)')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:423)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:355)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException
	at com.google.devtools.build.lib.rules.cpp.CppCompileActionBuilder.getActionName(CppCompileActionBuilder.java:250)
	at com.google.devtools.build.lib.rules.cpp.CppCompileActionBuilder.buildAndVerify(CppCompileActionBuilder.java:327)
	at com.google.devtools.build.lib.rules.cpp.CppCompileActionTemplate.createAction(CppCompileActionTemplate.java:120)
	at com.google.devtools.build.lib.rules.cpp.CppCompileActionTemplate.generateActionForInputArtifacts(CppCompileActionTemplate.java:91)
	at com.google.devtools.build.lib.skyframe.ActionTemplateExpansionFunction.compute(ActionTemplateExpansionFunction.java:83)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:346)
	... 4 more
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionTemplateExpansionKey{actionLookupKey=//antlr4/Native:Native com.google.devtools.build.lib.skyframe.BuildConfigurationValue$Key@2795597c false (392459051), actionIndex=1}' (requested by nodes 'antlr4/Native/_pic_objs/Native/antlr4/Native/generated.cc //antlr4/Native:Native com.google.devtools.build.lib.skyframe.BuildConfigurationValue$Key@2795597c false (392459051)')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:423)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:355)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException
	at com.google.devtools.build.lib.rules.cpp.CppCompileActionBuilder.getActionName(CppCompileActionBuilder.java:250)
	at com.google.devtools.build.lib.rules.cpp.CppCompileActionBuilder.buildAndVerify(CppCompileActionBuilder.java:327)
	at com.google.devtools.build.lib.rules.cpp.CppCompileActionTemplate.createAction(CppCompileActionTemplate.java:120)
	at com.google.devtools.build.lib.rules.cpp.CppCompileActionTemplate.generateActionForInputArtifacts(CppCompileActionTemplate.java:91)
	at com.google.devtools.build.lib.skyframe.ActionTemplateExpansionFunction.compute(ActionTemplateExpansionFunction.java:83)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:346)
	... 4 more

I tried with --features=parse_headers and --noparse_headers_verifies_modules as well, but to no avail.

Is there anything I can do to make compilation work? Or is is not (yet?) possible to use a tree artifact with .cpp and .h files?

[hlopko]

@marcohu You ask good questions, but not in a right place. We have to implement support for tree artifacts per-rule, and afaik java doesn't support them yet. Feel free to file an issue for that. Without much thinking about it I think cc_library should not accept zip files when it can accept tree artifacts. With that rules are more composable and focused.

Are there any plans to provide a truly uniform approach for the genrules use case? Can you elaborate (preferably on bazel-discuss, but here will do too, it's that only me will see it :)

Can you try running your C++ + antlr example with bazel@HEAD? We've done some improvements in this are recently.

[marcohu]

@mhlopko Thanks for your feedback.

The use case still fails with the current HEAD. I've opened #5092 with a minimal example to reproduce.

I'm certainly fine with the tree artifact approach. I opened #5093 as an umbrella for all languages. Or are specific issues for each language preferable?

My concern with the current C++ implementation is the necessity to use a language specific extension for the tree artifact. Ideally, this should not be necessary. With my antlr use case, this would mean the user has to specify the target language in the build file (while the generator could infer the language directly from the grammar) and would be forced to use different runs in order to generate code for different languages (otherwise even different languages could be generated in just one run). Not a big deal, but maybe something to consider. Not sure whether this is a topic for bazel-discuss.