Functionality Clarification for BWOB

[nickden3]

We have a large build (200k+ actions) and have noticed that clean BWOB builds with low cache hit rates (<70%) actually perform worse than no-cache builds, by a significant magnitude (2x). When we change the remote_download_outputs to all, then the no-cache build is the same performance wise as the cache-enabled one.

We are populating our remote cache, which is a simple nginx server backed by AWS S3, from a CI build on every commit to our default branch. These populate builds are incremental and are using remote_download_minimal as well. Should also note, it seems we see a higher cache hit % for read-only builds when the populate builds are fully clean.

Is this expected behavior and what are the recommendations around using remote_download_minimal vs remote_download_all? I've looked through the source code and forums and have not found any answers. Thanks.

Bazel Version: 7.0.0

[coeuvre]

This is not expected. It would be helpful if you can share the profiles for BwoB builds and no-cache builds.

[coeuvre]

Thanks for the profiles!

Looking at the bwob_cache profile, for many CppCompile actions, there is a gap between "check cache hit" and "CppCompile" spans:

For the other two, there aren't any gaps. That's probably why BwoB is slower in this case.

Now the question is what exactly did Bazel do within the gap? I looked at the code and it seems like the culprit is runfilesTreeUpdater.updateRunfiles. cc @tjgq

@nickden3: Can you try the BwoB build with flag --nobuild_runfile_links to check whether the build is faster?

[nickden3]

Hi @coeuvre , seems --nobuild_runefile_links was already being used. Should note, we're seeing this issue primarily on Windows builds.

Are there any other flags we should look into? Thanks.

[coeuvre]

Can you run a build with Bazel binary that includes commit 18c8839? I want to make sure it's indeed caused by runfilesTreeUpdater.updateRunfiles.

(The commit is included in the last_green, if using bazelisk, you can do USE_BAZEL_VERSION=last_green bazelisk ...)

[alinkonSF]

Hello, I'm picking this up for nickden3 while he's on vacation.

@coeuvre, I assume you want me to run with that commit and give you the trace?

[coeuvre]

@alinkonSF That's true. Thanks!

[alinkonSF]

Sorry for the long delay on the response; we had other work come up. We will get you the requested data at the start of February

[nickden3]

Hi @coeuvre , I reran our build using the bazel binary built from commit 18c8839. I sent the build profiles to you via email. Please let me know if you need any other info.

[coeuvre]

Thanks for sharing the profiles. It seems that the issue we saw from #20748 (comment) is gone, probably fixed by #20557.

With the new profiles:

1. Wall time for download all is 1738s.
2. Wall time for download minimal is 1826s, +5%.
3. Wall time for no cache is 2176s, +25%.

I also noticed that 1) and 2) have very low cache hit rate (probably < 10%), so the wall time variance between them might just because of local action executions.

#20557 will be shipped with Bazel 7.1.0. I suggest you try Bazel 7.1.0 once it is out (or a custom built Bazel with the fix) and report back if the issue still exists.