-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Protobuf repo recompilation sensitivity #7095
Comments
Can you double check that:
results in recompilation? What is the minimal content of the ~/.bashrc file that triggers the issue? What platform are you on? |
Looks like it's the Perhaps this is working as intended, or perhaps we can split it and use a set to dedup entries to normalize the path to the minimal necessary set. Is it possible to exclude |
$ uname -a
Linux bina 4.13.0-46-generic #51-Ubuntu SMP Tue Jun 12 12:36:29 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Hmmm then #6648 should actually help you (afaik). Can you try with bazel 0.21? |
or with 0.20 and with |
Yes! Tested on bazel 0.21.0 and indeed no longer recompiles with |
Awesome, thanks Paul. Can you pls share this solution on the mailing list? |
@pcj What toolchain do you use? Was it the Bazel's autoconfigured C++ toolchain? |
Also, on what platforms do you observe this? |
@hlopko I had observed it on linux & darwin before bazel 0.21.0. Now that I'm on 0.21 I'm really not seeing this anymore. For most projects I'm not using a special toolchain, so yes, the autoconfigured one. |
I believe that I found the root cause. It looks like this issue is not specific to C++, but that almost all actions (spawn actions, c++ compile, c++ link, java compile) in Bazel declare a dependency on |
That would make sense with our solution to this which was to launch bazel with |
@buchgr friendly ping |
@meisterT what's up? :) |
I have heard from bazel users that this is still a pain point. So I wanted to check in if you are working on a fix and when we can expect it to land? |
Yes I am working on a proposal to fix it. I don't want to make promises on when it will land. 1-2 quarters. |
I currently suffer from the protobuf targets rebuilding (a dependency of |
Fixes cockroachdb#71835. When switching between using `dev` and `bazel` raw, I kept seeing our C++ protobuf dependency getting recompiled (slowly). It appears that the bazel build for protobuf has a dependency on $PATH (see bazelbuild/intellij#1169 and bazelbuild/bazel#7095). Specifying `--incompatible_strict_action_env` pins PATH and avoids the build cache thrashing we were seeing before. Release note: None
Fixes cockroachdb#71835. When switching between using `dev` and `bazel` raw, I kept seeing our C++ protobuf dependency getting recompiled (slowly). It appears that the bazel build for protobuf has a dependency on $PATH (see bazelbuild/intellij#1169 and bazelbuild/bazel#7095). Specifying `--incompatible_strict_action_env` pins PATH and avoids the build cache thrashing we were seeing before. Release note: None
71632: sql: make sure pgwire bind always happens in a transaction r=otan a=rafiss fixes #70378 and maybe #64140 The approach of using a transaction matches what we do for pgwire Parse messages already. This is important to make sure that user-defined types are leased correctly. This also updated the SQL EXECUTE command to resolve user-defined types so that it gets the latest changes. Release note (bug fix): Adding new values to a user-defined enum type will previously would cause a prepared statement using that type to not work. This now works as expected. 71836: clusterversion: introduce 22.1 development versions r=celiala a=celiala Fix: #69828 Looking at past Version values for `Start{XX_X}`, I think the `Version` value for `Key: Start22_1` should instead be `Major: 21, Minor: 2, ...`. Start21_1 (from #70268): ``` // v21.1 versions. Internal versions defined here-on-forth must be even. { Key: Start21_1, Version: roachpb.Version{Major: 20, Minor: 2, Internal: 2}, }, ``` Start 21_2: ``` { Key: Start21_2, Version: roachpb.Version{Major: 21, Minor: 1, Internal: 1102}, }, ``` Release justification: Non-production code change. Release note: None 71856: bazel: skip recompilation when using dev+bazel r=irfansharif a=irfansharif Fixes #71835. When switching between using `dev` and `bazel` raw, I kept seeing our C++ protobuf dependency getting recompiled (slowly). It appears that the bazel build for protobuf has a dependency on $PATH (see bazelbuild/intellij#1169 and bazelbuild/bazel#7095). Specifying [`--incompatible_strict_action_env`](https://docs.bazel.build/versions/main/command-line-reference.html#flag--incompatible_strict_action_env) pins PATH and avoids the build cache thrashing we were seeing before. Release note: None Co-authored-by: Rafi Shamim <rafi@cockroachlabs.com> Co-authored-by: Celia La <celiala456@gmail.com> Co-authored-by: irfan sharif <irfanmahmoudsharif@gmail.com>
This is definitely still an issue - to be honest in projects that use Bazel I've never actually seen it "fixed" - I just got used to the fact that protobuf recompiles all the time. Hit this just today in
|
Seems like running gazelle can invalidate and cause a recompile for this package? |
@stevekuznetsov Can you write up repro instructions for cockroachdb w. gazelle please? |
@meisterT pinning down a direct reproducer is tricky. Cross-linked to an issue on the Cockroach project where devs talk about other targets getting recompiled, with some theories about why. |
I've tried to trace this cache fragility before and thought it might be protocolbuffers/protobuf#6886. However, IIRC that didn't solve it. When I've triggered this cache issue before with execution logging turned on, the action hashes between builds were identical but it was still rebuilding, suggesting it is not exclusively an env related issue. One consideration may be that protobuf is treated somewhat special by Bazel, in that there are native flags and rules that reference it. As such, this may be why protobuf behaves unlike any other external repo with regard to caching. Since the execution logs suggest there should be no rebuild, is there a way of tracing deeper within Bazel as to why it made a decision to rebuild? |
Presumably this would all go away if the proto rules were starlarkified as has been discussed for several years. Any update from the bazel team on this? |
cc @comius |
Any chance this can be bumped to a P1 please? It affects multiple people and has a direct impact on workflow because all builds and tests for those people are significantly slowed. In my case I think it may also be the root cause of Go IntelliSense bindings slowing down because they rely on |
I spent a little time this morning looking into why I was seeing very frequent protoc recompilations in cockroachdb/cockroach development and I believe that in my case might also be down to PATH changes as described above, but what was surprising to me is that this is a result of using the rules_go recommended editor configuration for my editor (VS Code). My editor is configured using the recommended configuration, which includes both a) pointing gopls at bazel and gopackagesdriver and b) setting Configuring the goroot seems more correct than letting vscode go invoke system go, so that when it runs However this means that when gopls is invoking gopackagesdriver and thus bazel, it is doing so with this modified Simulating this by hand, by running I guess there's not really new information here, insofar as protoc's sensitivity to PATH changes is already covered above, but what wasn't obvious to me at least when I started poking at this was that my PATH was changing so often, unbeknownst to me, just as a result of using the recommended editor configuration. |
yeah, this is really a bit of hidden knowledge that you learn through trial like you, but due to the sensitivity of A) only executing bazel in the same environment locally, e.g. always via VS Code, or always via your shell; or |
For what it's worth, the executions "via VS Code" here aren't voluntary ie me electing to trigger a build in my editor versus from my terminal -- they are the result of using the recommended editor configuration which changes its code navigation/inspection (via gopls) to be backed by bazel and gopackagedriver (which seems good!). These invocations happen "out of sight" and in the background so it wasn't obvious to me, at first, that they were what was thrashing PATH. |
i dont thonk the vscode config is the rooti ssue. we use vscode too, but point it to manually installed Go. so no changes to PATH or anything. yet merely afterbrunning gazelle to update a minor unrelated dependency rebuilds the entire protobuf stuff and thus the entire rest of the repo for some reason. |
@pdeva If you can minimize your repo to a reproducer you can share, I can investigate the rebuilds. |
Just one last update from my digging, just in case it can save anyone else some time if they're in a similar situation: I dug a bit more and realized that in our repo (cockroachdb/cockorach), we had developed particular sensitivity to PATH changes earlier this year when a developer added I have since changed our common dev configuration back to using a hardcoded PATH, based the default from foriegn_cc just with the addition of the new /opt homebrew location, and have not since heard additional reports of "spurious" protoc rebuilds 🤞. Again, this one is probably pretty obvious in hindsight -- having a tool config that constantly changes PATH and a build configuration that makes compilation depend on PATH implies constant recompilation -- but in isolation each of the steps involved to arrive at that situation seemed innocuous enough at the time. I just wanted to spell it out on the off chance it saves anyone else a little debugging time. |
I'm wondering if there are any updates on this. It's affecting our build as well. Would it be possible to implement a work around to expose a flag that allows for a pre-build version of the grpc compiler to be used instead of rebuilding it? Pantsbuild is using this approach. This has been open for 5+ years, and it seems quite tricky to reproduce and find the root cause for. A workaround might be a good interim solution |
I solved the rest of the build failures from bazelbuild#1618. Now multiple jobs are failing due to hanging scalac worker jobs as described in the PR description. - Proto Toolchainisation Design Doc https://docs.google.com/document/d/1CE6wJHNfKbUPBr7-mmk_0Yo3a4TaqcTPE0OWNuQkhPs/edit - bazelbuild/bazel: Protobuf repo recompilation sensitivity bazelbuild/bazel#7095 - bazelbuild/rules_proto: Implement proto toolchainisation bazelbuild/rules_proto#179 - rules_proto 6.0.0 release notes mentioning Protobuf Toolchainisation https://github.com/bazelbuild/rules_proto/releases/tag/6.0.0 It occurred to me to try adopting Proto Toolchainisation to see if that might resolve the issue. I've got it successfully generating the `@io_bazel_rules_scala_protoc` repo and registering toolchains for `@rules_proto//proto:toolchain_type`. However, it's currently dying because there's no toolchain registered for '@rules_java//java/proto:toolchain_type'. ```txt bazel build //src/... ERROR: .../src/protobuf/io/bazel/rules_scala/BUILD:4:14: in @@_builtins//:common/java/proto/java_proto_library.bzl%bazel_java_proto_aspect aspect on proto_library rule //src/protobuf/io/bazel/rules_scala:diagnostics_proto: Traceback (most recent call last): File "/virtual_builtins_bzl/common/java/proto/java_proto_library.bzl", line 53, column 53, in _bazel_java_proto_aspect_impl File "/virtual_builtins_bzl/common/proto/proto_common.bzl", line 364, column 17, in _find_toolchain Error in fail: No toolchains registered for '@rules_java//java/proto:toolchain_type'. ERROR: Analysis of target '//src/protobuf/io/bazel/rules_scala:diagnostics_proto' failed ERROR: Analysis of target '//src/java/io/bazel/rulesscala/scalac:scalac' failed; build aborted: Analysis failed ```
I solved the rest of the build failures from bazelbuild#1618. Now multiple jobs are failing due to hanging scalac worker jobs as described in the PR description. - Proto Toolchainisation Design Doc https://docs.google.com/document/d/1CE6wJHNfKbUPBr7-mmk_0Yo3a4TaqcTPE0OWNuQkhPs/edit - bazelbuild/bazel: Protobuf repo recompilation sensitivity bazelbuild/bazel#7095 - bazelbuild/rules_proto: Implement proto toolchainisation bazelbuild/rules_proto#179 - rules_proto 6.0.0 release notes mentioning Protobuf Toolchainisation https://github.com/bazelbuild/rules_proto/releases/tag/6.0.0 It occurred to me to try adopting Proto Toolchainisation to see if that might resolve the issue. I've got it successfully generating the `@io_bazel_rules_scala_protoc` repo and registering toolchains for `@rules_proto//proto:toolchain_type`. However, it's currently dying because there's no toolchain registered for '@rules_java//java/proto:toolchain_type'. ```txt bazel build //src/... ERROR: .../src/protobuf/io/bazel/rules_scala/BUILD:4:14: in @@_builtins//:common/java/proto/java_proto_library.bzl%bazel_java_proto_aspect aspect on proto_library rule //src/protobuf/io/bazel/rules_scala:diagnostics_proto: Traceback (most recent call last): File "/virtual_builtins_bzl/common/java/proto/java_proto_library.bzl", line 53, column 53, in _bazel_java_proto_aspect_impl File "/virtual_builtins_bzl/common/proto/proto_common.bzl", line 364, column 17, in _find_toolchain Error in fail: No toolchains registered for '@rules_java//java/proto:toolchain_type'. ERROR: Analysis of target '//src/protobuf/io/bazel/rules_scala:diagnostics_proto' failed ERROR: Analysis of target '//src/java/io/bazel/rulesscala/scalac:scalac' failed; build aborted: Analysis failed ```
As discussed on the mailing list https://groups.google.com/forum/#!topic/bazel-discuss/UKLK7Y-jsW4 the protobuf repo is notoriously sensitive to recompilation. Presumably something in the environment used in the action key for protobuf C++ compilation leads to cache invalidation and hence recompiling protobuf from source.
Given protobuf libs are often base dependencies for many others, leads to degraded user experience.
Repro steps include taking any workspace that has a proto dependency and
source ~/.bashrc
, watch protobuf recompile.Observed up to and including bazel 0.20.0.
/assign @hlopko
The text was updated successfully, but these errors were encountered: