bazel: the captured cmake dependency does not build efficiently

[knz]

Describe the problem

When running bazel for the first time after a c++ compiler update, or a version change etc, it needs to re-build its embedded (captured) copy of cmake again.

However, the build process for cmake is veeeerrrryyyyyyy looooooonnnnnngggggg
(upwards of 10 minutes)

The main problem is that the build flags do not include -j or something similar to parallelize the build, so we're seeing hundreds of c++ files compiled one after another.

To Reproduce

Build from scratch without a cached pre-built cmake.

Expected behavior

The cmake builds uses all 32 cores on my computer.

Additional data / screenshots

htop output:

Epic CRDB-8036

[rickystewart]

The behavior is actually "correct" inasmuch as we should only be using multiple CPU's if we can have Bazel parallelize the workload, which it can't meaningfully if sub-tasks are arbitrarily using as many cores as they want under the scenes.

In terms of making this faster I'm more interested in patching rules_foreign_cc to download a pre-bootstrapped cmake from an appropriate place rather than building from scratch.

[knz]

it can't meaningfully if sub-tasks are arbitrarily using as many cores as they want under the scenes.

Ok but how do you explain that the cmake build inside the bazel sandbox does not parallelize, whereas it does when run outside of the sandbox?

I'm more interested in patching rules_foreign_cc to download a pre-bootstrapped cmake from an appropriate place rather than building from scratch.

That would be lovely.

[rickystewart]

it can't meaningfully if sub-tasks are arbitrarily using as many cores as they want under the scenes.

Ok but how do you explain that the cmake build inside the bazel sandbox does not parallelize, whereas it does when run outside of the sandbox?

Not really sure what this question means. Doesn't your own observation answer this question?

The main problem is that the build flags do not include -j or something similar to parallelize the build, so we're seeing hundreds of c++ files compiled one after another.

[rickystewart]

bazel-contrib/rules_foreign_cc#329 for reference on the overarching issue here.

[knz]

ok so the root cause is that bazel does not implement the job pool mechanism that GNU make supports (and I think cmake too). That's pretty unfortunate.

In any case, the issue at the top remains. Building cmake costs 5-10 minutes and is currently responsible for 60% of the run time of the "Bazel CI" target in TC.

[irfansharif]

I'm seeing this too for local builds. The majority of the time spent in a bazel build is actually spent building cmake from scratch, which is surprising.

[2,367 / 3,189] BootstrapGNUMake external/rules_foreign_cc/toolchains/make [for host]; 379s darwin-sandbox ... (16 actions running)

[knz]

Is there a way to tell Bazel to use the host cmake instead of baking a new one? At least for local builds?

[rickystewart]

Yeah, this should do it:

diff --git a/WORKSPACE b/WORKSPACE
index 4066d7f09d..a232d4355d 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -172,7 +172,11 @@ git_repository(
 
 load("@rules_foreign_cc//foreign_cc:repositories.bzl", "rules_foreign_cc_dependencies")
 
-rules_foreign_cc_dependencies()
+rules_foreign_cc_dependencies(
+    register_built_tools = False,
+    register_default_tools = False,
+    register_preinstalled_tools = True,
+)
 
 # Load custom toolchains.
 load("//build/toolchains:REPOSITORIES.bzl", "toolchain_dependencies")

Might be a way to do it with command-line options instead of updating the workspace, not sure what that is yet though.

[knz]

for now, I'll take that as a workaround. thank you!

(Still would be interesting to lower the time of the CI target, but for that I support your idea to download a pre-built binary)

[rickystewart]

(Still would be interesting to lower the time of the CI target, but for that I support your idea to download a pre-built binary)

I took a look and rules_foreign_cc already kind of supports this, but 1) I'm not sure if they've verified FreeBSD compat and 2) they haven't registered pre-built make binaries to download (only cmake and ninja, not that we're using the latter). Probably would be a pretty small work item to fix this all told.

[knz]

I'm not sure if they've verified FreeBSD compat

I can use your workaround if need be