Bazel Rules for Cuelang

[gravypod]

I was wondering if there was a canonical set of rules for using cuelang in a bazel build environment. For my usecase this would be helpful for reducing the tooling a new engineer will need to learn and allow for better integration with things like tilt bazel run //deployments:development would be very nice.

I think this also makes sense to mainline in cuelang as, from the README.md, code generation from CUE is a planned feature. The ability to have a cue_library and automatically link generated code into projects of different languages would be very helpful.

[mpvl]

Can you give a more concrete example of how things look like for what you had in mind?

[gravypod]

@mpvl most languages contain some form of rules_<language> that supply <language>_library and <language>_binary.

Jsonnet, for example, has a rules_jsonnet which supplies some common functionality that you might want to do:

1. jsonnet_library: collection of jsonnet files that become importable
2. jsonnet_to_json: Convert jsonnet libraries into json data

Something that might work for cuelang would likely be:

1. cue_library: Making sources visible to your build
2. cue_gen_proto: Generate .protos from cue definitions
3. cue_gen_openapi: Generate openapi from cue definitions
4. cue_gen_yaml: Gen yaml
5. cue_gen_json: Gen json
6. cue_gen_kube_yaml: Maybe include some magic that allows higher level abstractions that could integrate with other bazel rules (rules_docker for example). Please.build has an example of this working.

Some of these might be better to implement outside of a rules_cuelang (specifically 5)

[mpvl]

Gotcha. This would have to be build int the bazelbuild Github repo.

Also, I'm quite unfamiliar with bazel and this should ideally be build by someone that is using this in practice. I am happy to help or guide the person doing this, though.

[rudolph9]

Though I'm not familiar with this bazel rules either, a step in this direction may be to support builds using tinyGo. I may be mistaken but I believe some of the same limitations around go modules are shared between a bazelbuild and a tinyGo build. A tinyGo build would also likely reduce the overhead of running cue and make the WASM target much lighter (relate wasm/npm ticket)

[davidzchen]

@mpvl I'm a Bazel contributor and the original author of bazelbuild/rules_rust, bazelbuild/rules_jsonnet, and bazelbuild/rules_d. I'm interested in writing Bazel rules for Cue.

Since Bazel Cue rules will depend on the cue executable, my understanding is that we would need to add BUILD files for building Cue (using bazelbuild/rules_go). I think bazelbuild/bazel-gazelle will be helpful for generating and updating BUILD files.

I think the rules that @gravypod listed is a good start. For data validation, I am thinking we can have bazel test rules (perhaps cue_validate_yaml_test or something similar).

I will have to do more reading on Cue's other use cases, but if aren't objections, I can write a short proposal and start working on a proof of concept of building Cue with Bazel and an initial prototype with cue_library, a couple of rules for generating JSON and YAML, and a validation test rule.

WDYT?

[seh]

If you didn't feel like arranging for building cue in these rules, you could instead download built executable files using http_archive, since this project publishes such files for three platforms: macOS, Linux, and Windows.

[gravypod]

@seh would be an option but http_archive and other workspace rules are a little cumbersome. If build files and rules were in repo you could vendor this project in your company's third_party very easily. You could also use git_repository to refer to any commit.

[seh]

I use http_archive like that in four Bazel packages I maintain, for Packer, Terraform, ct, and kubecfg. It works just fine, without much fuss, but I understand that it's not quite "the Bazel way." However, I wouldn't want to hold up development of rules that address using cue on developing rules to build cue.

[davidzchen]

bazelbuild/bazel-gazelle can be used to automatically generate BUILD files, and it also adds all the go_repository rules for external dependencies to the WORKSPACE file.

I was able to generate WORKSPACE dependencies and BUILD.bazel files for Cue with Gazelle and successfully build //cmd/cue with the generated BUILD files:

I still need to verify that all the tests run successfully, but I think I can send a PR for this soon.

[davidzchen]

FYI the changes are in the bazel branch in my fork: https://github.com/davidzchen/cue/tree/bazel

[davidzchen]

Some remaining issues to resolve from this first pass:

• Correctly embed testdata files in tests, such as for //cue/load/import_test.
  • BUILD.bazel files are being generated in some of the testdata directories. (https://github.com/bazelbuild/rules_go#how-do-i-access-testdata), and these are likely extraneous and should be removed
  • Testdata be instead using the data attribute per the [rules_go documentation]
• Set up protobuf dependency to WORKSPACE for proto-related tests under //encoding/protobuf
• Add the necessary Gazelle directives to correctly generate build files, taking into account the above.

[davidzchen]

Most tests are now passing:

[davidzchen]

Sorry for the delay. I’ve been busy, but I have resumed working on this. There are some issues with getting some of the tests run by Bazel to correctly find the testdata files. Hoping to get the PR out within the next week.

[steeve]

Since Bazel Cue rules will depend on the cue executable, my understanding is that we would need to add BUILD files for building Cue (using bazelbuild/rules_go). I think bazelbuild/bazel-gazelle will be helpful for generating and updating BUILD files.

I think it's perhaps better to just download the cue binaries maybe? That's would remove the dependency to rules_go. Although proto import isn't exposed in the CLI.

If you didn't feel like arranging for building cue in these rules, you could instead download built executable files using http_archive, since this project publishes such files for three platforms: macOS, Linux, and Windows.

I would +1, but since importing from proto isn't exposed in the CLI, we need to compile it anyhow :(

Also, I'm not sure adding bazel files in this repo is the way to go unless Cue maintainers actively use them. Worst would be for those build files to be unmaintained.

[tnarg]

I've created an initial set of Bazel rules for cue: https://github.com/tnarg/rules_cue. I'd love to get your feedback on them. They include a cue_binary rule which performs a cue export, and a cue_library rule for bundling a cue package. The rules also include a gazelle extension which is used by the repo itself to generate BUILD.bazel files for the examples.

[steeve]

@tnarg super nice! saw a few things but overall really nice!

[steeve]

@tnarg opened a few issues! very impressed overall! I would have opened PRs but I'm busy on something else at the moment. Great job.

[cueckoo]

This issue has been migrated to cue-lang/cue#163.

For more details about CUE's migration to a new home, please see cue-lang/cue#1078.