[Self-Filed] Bzlmod support

[cpsauer]

Bazel's working on a new external dependency management system called Bazelmod, arriving in 6.0 and by default in 7.0, replacing and deprecating the WORKSPACE.

Please chime in if you're looking to use bzlmod (or even help us support it!) so we can support the new external dependency syntax at the right time.

Cheers!
Chris

[kylepl]

Any further updates on when this may be available? Or an estimate of effort?

As someone setting up Bazel usage with bzlmod, this sounds great to use - especially if it allows for easy source-code viewing of bzlmod packages pulled in via http archives.

[cpsauer]

Hey, Kyle! Honestly, we haven't really spent much time scoping bzlmod yet. This issue was meant to track interest in order to figure out the right time to add bzlmod support. I see there are now lots of 👍🏻; thanks for commenting and bringing me back here.

For my prioritization and understanding: Is WORKSPACE instead of bzlmod blocking you (or others reading this)? Are there important things it's getting you that WORKSPACE file does not? Are there things about, e.g., the new canonical workspace stuff, that are breaking features for you?

My mental state, if useful as a diff base. Please tell me if I'm wrong or if there are other things I should be thinking about!

• I'd think the WORKPACE setup would still work alongside experimental/opt-in bzlmod.
• I'd guess we'd get you easy source-code viewing of bzlmod modules through the external symlink, assuming that bzlmod still brings in external workspaces in pretty much the same way.
• I'm guessing that we're still early enough in bzlmod's experimental/opt-in lifespan that we shouldn't focus on it as a distribution mechanism just yet for this tool--but that we should start trying to fix any issues from new bazel structure bzlmod creates when bringing in other dependencies. I'm thinking this, since I hadn't yet seen any of our co's deps recommending it, google/bazelbuild repos included, plus Renovate doesn't support it yet (👍🏻, comments, PRs over at Renovate would be helpful!). So it didn't seem like we should switch or advise doing so just yet--and it seems like it'd be extra release work vs living at HEAD, abseil-style. But we do want to start working on supporting new dependencies being brought in via bazelmod if there are issues there, since we want to make everyone more productive and not block anyone from starting to migrate.
• More generally, I'm wondering how (if ever) they'll actually get people off using WORKSPACEs, or whether the two might end up coexisting long term. Being able to easily wrap non-pre-bazeled code seems quite valuble.

Thanks!
Chris

[curoky]

Perhaps this repo need a tag, so that I can make a PR on bazel-central-registry.

https://github.com/bazelbuild/bazel-central-registry/compare/main...curoky:bazel-central-registry:main?expand=1

[aaronmondal]

Hi, I've been using bzlmod for quite some time in rules_ll and I'm the one (slowly 😅) adding bzlmod support to LLVM. In case it helps:

• I'd think the WORKPACE setup would still work alongside experimental/opt-in bzlmod.

Yes it does. Even register_execution_platform and register_toolchains work and apparently interact with each other between WORKSPACE.bazel and MODULE.bazel.

• I'd guess we'd get you easy source-code viewing of bzlmod modules through the external symlink, assuming that bzlmod still brings in external workspaces in pretty much the same way.

I think so, but the file structure might have changed. Bzlmod exclusively uses the (new?) "canonical" path schemes which can sometimes look a bit strange like .../external/<bzlmod_dep_name>~<version>~<extension_name>~<repo_name> for paths. This is a non-issue in starlark, but might make it hard to write custom path parsers.

• I'm guessing that we're still early enough in bzlmod's experimental/opt-in lifespan that we shouldn't focus on it as a distribution mechanism just yet for this tool--but that we should start trying to fix any issues from new bazel structure bzlmod creates when bringing in other dependencies. I'm thinking this, since I hadn't yet seen any of our co's deps recommending it, google/bazelbuild repos included, plus Renovate doesn't support it yet (👍🏻, comments, PRs over at Renovate would be helpful!).

I'm not familiar with the codebase for this tool, but I'd guess that adding bzlmod as distribution mechanism is probably a lot easier than adding parsing for modules. Basically, you can just copy-paste most of the previous repos.bzl etc files to an extensions.bzl and things will more or less work. E.g. the current state of the LLVM bzlmod support is here.

So it didn't seem like we should switch or advise doing so just yet--and it seems like it'd be extra release work vs living at HEAD, abseil-style. But we do want to start working on supporting new dependencies being brought in via bazelmod if there are issues there, since we want to make everyone more productive and not block anyone from starting to migrate.

It definitely feels a bit experimental. Stardoc is just getting support for it, but from an in-organization perspective it's amazing how clean an user-friendly end user MODULE.bazel files are compared to WORSPACE.bazel files. Consider this example file which "just makes things work" without manually registering any toolchains or execution platforms since that's already handled by the bzlmod module automatically.

• More generally, I'm wondering how (if ever) they'll actually get people off using WORKSPACEs, or whether the two might end up coexisting long term. Being able to easily wrap non-pre-bazeled code seems quite valuble.

This is of course highly opinionated, but IMO once you go bzlmod you won't want to go back. The dev experience is incredibly nice once you have it set up. It took 3(!) attempts to make things work well with custom registries etc (we have a bazel-eomii-registry, bazel-eomii-registry-deprecated and bazel-eomii-registry-also-deprecated 🤣) and a bit of tinkering with versioning (ended up with <version>.bcr.n for our modules, where n is an additional identifier to separate different module versions of the same underlying repo), but once things WAI, they really WAI.

I don't think I'd have time to create a PR here myself, but I'd be happy to provide feedback on PRs etc if it helps you.

[cpsauer]

Thanks guys.

Based on that, I think the code, as is, will already play nicely with modules (aka external workspaces) brought in via bzlmod, since we don't assume anything about the name of those workspaces (in the path, above). That is, @kylepl, I think we'll already help you accomplish your goal of "allow[ing] for easy source-code viewing of bzlmod packages." If that's wrong, again, please say so we can work together to get it fixed asap.

My working hypothesis remains that we should probably continue to recommend bringing this tool in via the WORKSPACE just until there's Renovate update support for bzlmod, since I'm assessing that WORKSPACE copy-paste integration is pretty easy and already not much of a barrier--but that encouraging updates is pretty important to getting the best experience for everyone per unit effort. But, as usual, if you think I've misassessed, please do say. Or if someone's knows how to make bzlmod automatically pickup the latest commit, with GitHub actions, say--and would be willing to help with a PR--we could continue to live at head while supporting it alongside in the meantime.

Cheers,
Chris

[Attempt3035]

Hi all,
I'm just wondering if anyone could provide some instruction on how to use this project with bzlmod, as I understand, it should still work even though it's using the WORKSPACE system, but it would be very helpful to have a couple of pointers of how to get this working with a newer bzlmod project as I'm a bit lost on how to add it through using an extension. I'd be happy to transpose this into a PR (once I get it working) to add to the README for anyone else who comes looking for this answer too!

EDIT:
I've figured it out - Polishing up the solution, will link here for others shortly...

[cpsauer]

Ooh great! Thanks for your efforts, Luke.

For my learning: I was thinking people would just use the WORKSPACE snippet from their WORKSPACE during the migration period. Anything I should know there? (I think both work together and am guessing it'll be a long time (if ever) until they get rid of the WORKSPACE.)

[Attempt3035]

Yes, according to the readme here, both WORKSPACE and MODULE.bzlmod can work together, although personally, I'm setting up a new project so would like it all to be in the new system as that is set to become default in Bazel v7 (although I'm sure workspace will still be around for a long time).

TO ADD bazel-compile-commands-extractor TO BZLMOD VIA AN EXTENSION:

1. Create an extensions.bzl file in the same directory as your primary MODULE.bazel file
2. Populate extensions.bzl with the following:

# extensions.bzl
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

def _hedron_compile_commands_extension_implementation(ctx):
    http_archive(
        name = "hedron_compile_commands",
        url = "https://github.com/hedronvision/bazel-compile-commands-extractor/archive/ac6411f8f347e5525038cb7858db4969db9e74f2.tar.gz",
        strip_prefix = "bazel-compile-commands-extractor-ac6411f8f347e5525038cb7858db4969db9e74f2",
        sha256 = "f03ed383f1093b5960b37648c4fc5ae2b3d98050509bc2e11719afc8b1fd3b3a",
    )

hedron_compile_commands_extension = module_extension(
    implementation = _hedron_compile_commands_extension_implementation,
)

3. Add this line to your MODULE.bazel file:

use_repo(use_extension("//:extensions.bzl", "hedron_compile_commands_extension"), "hedron_compile_commands")

(Or alternatively something like this if you find it neater:)

hedron_compile_commands = use_extension("//:extensions.bzl", "hedron_compile_commands_extension")
use_repo(hedron_compile_commands, "hedron_compile_commands")

That's as simple as I could get the process, the docs here provide an extended way of doing this under the "Fetch external dependencies with module extensions" section, which I had working in the extended form, but now it's not and I'm not sure what I broke😅

Anyway, the extended form just broke it down into a repositories.bzl file that had the http download and then used that function in the extensions.bzl, which was then included in the main repo.

I do wonder if we could have a MODULE.bazel file added to this project that would work alongside the WORKSPACE file, they should both be fine to work together, and Bazel will simply automatically choose preference of the MODULE.bazel if bzlmod is enabled on their setup - this would make it easier to get included without needing the extra extensions file and shouldn't be a hard fix!

[Attempt3035]

I've created a PR with a permanent fix, all it took was adding a MODULE.bazel and extensions.bzl file to the repo, all functionality now works with bzlmod smoothly via a simple inclusion.

This works much better than the hacky fix above, as it actually solves the root support problem by having the complimentary MODULE.bazel file available in the repo. It also means the commands are executed by the repo's extension file itself, rather than the user doing it. The above hack also didn't have support for running the workspace configuration method, but the PR method does, so it's a seamless and fully functioning transition from the WORKSPACE system.

See PR #146

[cpsauer]

Woo! So Luke got found a great way for us to get bzlmod support out for all you early adopters--and it's hot off the presses. Please check out the bzlmod install path, added to the README, and tell us whether it's working for you!

[cpsauer]

We'd love also your double checking that Renovate properly upgrades the bzlmod version, letting us know either way and filing a discussion over at Renovate if not.