build: Add the parallel feature, set -j arg based on NUM_JOBS

[ParkMyCar]

This PR speeds up builds of protobuf-src for me from ~5 minutes to ~1 minute.

There's probably a higher level issue that I haven't dug into here yet, the actually cmake command that gets run, without this PR, is:

MAKEFLAGS="-j --jobserver-fds=8,9 --jobserver-auth=8,9" cmake --build . --target install --config Debug

This should parallelize the build with Cargo's jobserver (I think?), but something seems to go wrong and at least locally, after ~30 seconds I only see one instance of clang running.

With this PR the cmake command is updated to:

MAKEFLAGS="-j --jobserver-fds=8,9 --jobserver-auth=8,9" cmake --build . --target install --config Debug -- -j12

Further, looking at the logs from cmake this seems to disable jobserver mode:

[protobuf-src 2.1.0+27.1] make: warning: -jN forced in submake: disabling jobserver mode.

This isn't ideal, but something seems to be going wrong with Cargo's job server which is seriously slowing down builds.

[def-]

My main concern with just parallelizing a lot is that this can cause OoMs.

I think cmake might just ignore the MAKEFLAGS. How about adding the entire -- -j --jobserver-fds=8,9 --jobserver-auth=8,9 at the end instead?

[benesch]

Hrm, seems like some upstream issues about this:

• Parallel builds break when combined with cc-rs using the "parallel" feature rust-lang/cmake-rs#172
• New jobserver type in GNU Make 4.4 rust-lang/jobserver-rs#47

Not sure what the right fix is, but concur with Dennis that disabling the jobserver and allowing parallelism has led to OOMs in CI during builds in the past.

[petrosagg]

Is there a way to have this for builds on dev machines? It's rare that this crate needs to be rebuilt but not THAT rare, especially if I'm jumping around branches etc

[benesch]

Is there a way to have this for builds on dev machines? It's rare that this crate needs to be rebuilt but not THAT rare, especially if I'm jumping around branches etc

It's still dicey. We've seen compiling Materialize OOM on dev machines in the past when we haven't been careful to let one single jobserver be responsible for managing all the parallelism.

The jobserver protocol isn't all that complicated. I'm hopeful that it might be possible to track down whatever the root cause here is without too much trouble.

[ParkMyCar]

Concerns about memory usage totally makes sense! I'm going to close this out for now since this PR isn't the right approach, but also I don't have the cycles at the moment to investigate the upstream jobserver issue. As a workaround folks building Materialize can use Bazel which handles this more gracefully.

[kesyog]

FYI there's a great chance this issue would be fixed by rust-lang/cmake-rs#229. One workaround in the meantime is to set the CMAKE_BUILD_PARALLEL_LEVEL environment variable based on NUM_JOBS (or go with what this PR did, which is equivalent). It can be limited to OSX builds only to help assuage OOM fears.

[benesch]

Ahh, nice, thanks for tracking that down, @kesyog! What's the behavior on macOS with your PR? Does CMake pass -jNCPUs or something like that?

[kesyog]

Yeah, pretty much exactly. So there’s still potential for extra memory usage with this change, but it is at least for macOS only. And the number of jobs could be limited if that caused issues for macOS builders (see below).

A few pedantic clarifications:

• The -j value passed will follow whatever Cargo uses so it could optionally be limited to fewer than the number of CPU’s via https://doc.rust-lang.org/cargo/reference/config.html#buildjobs
• Even on macOS, the jobserver information will be passed to make rather than using -j if the jobserver uses the newer named pipe approach. However, this would only occur in the super niche case where cargo were launched from within some other build environment that had already created this type of jobserver.

[benesch]

Makes sense! So @ParkMyCar I think we should just update to the next version of cmake-rs (whenever @kesyog's PR makes it into a release) and hopefully that fixes folks' issues here. 🤞🏽