Switching back and forth from terminal to IDE requires rebuilds

[gmishkin]

Whenever I run a bazel build in the terminal and then switch to intellij and resync the project, it does a full rebuild. It says the options have changed so it's discarding the cache.

Looking at the options it sets, --tool_tag stands out as an obvious one that is in the affects_output section of the Bazel CLI docs, so could that be causing it? There are others as well.

How necessary are these and are there any workarounds or improvements that could be made to make switching back and forth a little easier/faster?

If it sounds like I'm talking crazy talk because this isn't happening for you, I can give more details about the project, if you point me in the right direction to start.

[chaoren]

--tool_tag should not be the culprit.

Do you have any custom flags in the .bazelproject file?
Do you have a different JAVA_HOME on the command line vs in the IDE?

[gmishkin]

No, and I'm using @bazel_tools//tools/jdk:absolute_javabase as the --[host_]javabase in a bazelrc to try to avoid the issue.

[promiseofcake]

I have noticed in In Goland (pre Bazel Bazel 0.22) it's historically been due to IJ PATH creep from within the IJ terminal.

[chaoren]

How are you launching the IDE? If you launch it via command line instead of the launcher icon, then it should inherit the same environment variables as bazel on the command line.

If no other flags are added, then it's probably caused by different environment variables.

[promiseofcake]

How are you launching the IDE? If you launch it via command line instead of the launcher icon, then it should inherit the same environment variables as bazel on the command line.

Very interesting, we have been launching it through the Icon / JetBrains manager. I will give this a try.

[kien-truong]

I also had this issue when launching IntelliJ from outside the terminal. Starting IJ from the terminal fixed it.

We checked the environment variables, but there aren't any significant difference; so we are still clueless about the root causes.

[kien-truong]

It was indeed due to the difference in the PATH variable that caused the issue for me. I have a few terminal plugins that modify that variable, but only for interactive shell. Resetting PATH to the same value as IntelliJ's also fixed the issue.

[promiseofcake]

It appears that there are specific things still added by the Goland IJ terminal that are not present in the external shell's $PATH when launched via the command line. We have modified our workflow so this is not a problem and I believe some newer-bazel defaults regarding action-env may have resolved this.

[gmishkin]

I found a situation where JAVA_HOME actually was different, sorry. Making sure this was unset in both places made it keep the cache. However as I mentioned, I am using :absolute_javabase as my --javabase and --host_javabase specifically to avoid issues like this, so it doesn't seem like it should matter what that environment variable is set to.

So do you think it's actually a Bazel bug that's causing the invalidation if the environment changes like this since it shouldn't actually matter?

[mumbleskates]

This also happens to me in C++ projects. Unless I use the terminal emulator in the tool window inside clion, bazel always discards cached results when switching between terminal and the IDE.

There are diffs in the env:

Unset in system terminal: APPDIR APPIMAGE GIO_LAUNCHED_DESKTOP_FILE GIO_LAUNCHED_DESKTOP_FILE_PID OWD TERMINAL_EMULATOR TOOLBOX_VERSION

Unset in IDE terminal: COLORTERM GNOME_TERMINAL_SCREEN GNOME_TERMINAL_SERVICE VTE_VERSION

Set in both but empty in IDE terminal: GTK_MODULES

Set to different value: OLDPWD (pretty sure this is fine 😛)

When I copied the six env values other than OWD in the "Unset in system terminal" category above into a system shell, it stops having this rebuild problem.

... and now that I've done that once, even in a new terminal and across calls to clean --expunge I cannot seem to reproduce the problem at the moment.

It's been an ongoing and especially insidious problem because some of my build rules call into existing non-bazel build processes and take a lot of CPU time to finish, yet will typically rerun every single time I go back and forth between system terminal and IDE sync.

I'll see if I can figure out how to make it start reproducing again.

edit: No dice after a reboot, things are continuing to work flawlessly as they have never before. I will check another dev machine.

[mumbleskates]

Yes, it's still reproducible from the other machine. 200+ rules rebuild each iteration going back and forth between syncing in the IDE and building in the system terminal. It's still happening, it seems to happen in the default configuration, and it's very unfun.

[mumbleskates]

On yet another machine, even switching back and forth between IDE sync and bazel build in the IDE terminal triggers extensive rebuilds.

[chaoren]

bazel build in the IDE terminal

The IDE terminal still loads your rc script as an interactive shell. Does that machine have additional environment variables set there?

What's the exact output that bazel gives when it tries to rebuild? Does it tell you which flag/variable changed?

[mumbleskates]

As far as I've seen there is no message indicating that cache is being discarded, it simply rebuilds a lot of things as if they were not there. I've looked pretty closely.

The only extra things in my rc file are like PS1, GIT_PS1_SHOWDIRTYSTATE, and sourcing bazel-complete.bash.

I'm not entirely sure, but the items that are getting rebuilt seem to be, or be downstream of, genrule rules. There are some in gflags, some in protobuf, some in my buildfile for yasm, and all of libsodium is a single genrule; these seem to get rebuilt every time. Currently on my laptop (probably the best example, as it's broken and has only publicly available versions of bazel and ubuntu) they rebuild even when going back and forth between the IDE sync and the IDE terminal.

[chaoren]

You don't even get this message?

WARNING: Running Bazel server needs to be killed, because the startup options are different.

[mumbleskates]

Not at all, no.

[mumbleskates]

Here's some example text I get from the Blaze Console during a sync (this is after syncing, bazel build ... in terminal, syncing, bazel build ... in terminal etc.):

Updating VCS...
Running Bazel info...
Command: bazel info --tool_tag=ijwb:CLion --curses=no --color=yes --experimental_ui=no --progress_in_terminal_title=no --

Command: git diff --name-status --no-renames 60d0b2fa4c6fa9a68761ec53d79e71ee6af12316

Computing VCS working set...
Your working set is empty
Sync targets from project view:
  //...

Building blaze targets...
Command: bazel build --tool_tag=ijwb:CLion --keep_going --build_event_binary_file=/tmp/intellij-bep-a27ed71a-c469-4792-b427-d8e138014d7f --nobuild_event_binary_file_path_conversion --curses=no --color=yes --experimental_ui=no --progress_in_terminal_title=no --aspects=@intellij_aspect//:intellij_info_bundled.bzl%intellij_info_aspect --override_repository=intellij_aspect=/home/widders/.CLion2019.1/config/plugins/clwb/aspect --output_groups=intellij-info-cpp,intellij-info-generic,intellij-info-py,intellij-resolve-cpp,intellij-resolve-py -- //...

INFO: Loading complete.  Analyzing...
INFO: Loading package: @local_config_sh//
INFO: Found 161 targets...
INFO: Building...
[3 / 6] Executing genrule @com_github_gflags_gflags//:gflags_declare_h
[447 / 464] Executing genrule //third_party/libsodium:build
INFO: From Executing genrule //third_party/yasm:YASM-VERSION:
1.3.0
INFO: From Executing genrule //third_party/yasm:YASM-VERSION [for host]:
1.3.0

[mumbleskates]

If there's any additional information I can provide I'm happy to do so. The @com_github_gflags_gflags//:gflags_declare_h rule is probably a good starting point...

[chaoren]

We should probably move this over to http://github.com/bazelbuild/bazel, so they can help us figure out what caused the rebuild.

[mumbleskates]

AH! It's looking like there's a strong possibility that garbage left behind by python virtualenv scripts in $PATH are causing genrule builds to be discarded, and likewise every single thing that transitively depends upon them as well.

It's a bit odd that it doesn't get logged, but it does look like this is probably the underlying cause. I will check on the laptop when I get home; hopefully it holds true there as well.

[mumbleskates]

Nope, no luck. I've done what I can to ensure that PATH is exactly the same in both cases and it is still rebuilding genrules on the laptop.

[mumbleskates]

..........no, I realized the last thing I'd forgotten today.

I have a script with CC=clang bazel $@ configured as the bazel binary in and an alias for the same in the terminal (in lieu of toolchains, which are a nightmare to set up), and I had forgotten one of these.

I could probably file a feature request for genrules to log why they are being rebuilt.

[steinar]

I had this problem and found it was indeed the environment $PATH variable.

By inspecting external/@local_config_sh.marker under Bazel's working directory I noticed the value for ENV:PATH changed as I went between IntelliJ and terminal. This appears to trigger a full rebuild. Running IntelliJ from the terminal did not work for me (MacOS Mojave + iTerm + zsh + IntelliJ 2019.1).

Moving my custom path from .zshrc to /etc/paths.d/ made it consistent.

[mumbleskates]

Yes, $PATH is one of the rebuild-triggering vars for essentially any genrule, which are often upstream of a lot of expensive compilation.

[natros]

How can I see the path? Bazel info does not show any path. Is there a flag to dump all the info that is used to detect the environment changed.

[mumbleskates]

If you run a build command with -s (or --subcommands, the long form) it will show you all the commands that are run, which will look like, as an example:

SUBCOMMAND: # //third_party/yasm:genstring_license [action 'Executing genrule //third_party/yasm:genstring_license']
(cd /home/widders/.cache/bazel/_bazel_widders/abcdefabcdefabcdefabcdefabcdefab/execroot/cc && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/host/bin/third_party/yasm/genstring license_msg bazel-out/k8-fastbuild/bin/third_party/yasm/libyasm/frontends/yasm/license.c third_party/yasm/libyasm/COPYING')

...which as you can see shows explicitly setting PATH in a subshell before executing the command.

[mumbleskates]

Alternative, which I have actually done: just add env | sort > /tmp/whatisthegenruleenv ; to the genrule command and cat /tmp/whatisthegenruleenv afterwards.

[prestonvanloon]

Has anyone resolved this? I see the same issue with building protobuf when switching between goland and terminal.

[prestonvanloon]

fwiw: I didnt experience this in goland 2019.1.x

[chaoren]

So there's the --explain and --verbose_explanations flags that will tell you why the rebuild occured.

Just run bazel build --explain=/tmp/explanation.txt --verbose_explanations ..., and take a look at /tmp/explanation.txt, you'll see something like:

Executing action 'Executing genrule //path/to:target': Effective client environment has changed. Now using
  PATH=/usr/local/bin:/usr/bin:/bin:...
.

It doesn't tell you what your previous PATH was, so that's kind of annoying, but you can run with the two flags in both environments and then compare the differences.

[prestonvanloon]

Why is the intellij bazel plugin using a different path? I don't understand the resolution.

[chaoren]

The IDE inherits its PATH from the desktop environment. Your command line is likely the one using a different PATH. We can't control how you modify your PATH in your rc scripts. If you look at what's actually getting added/removed from your PATH, then it could help you narrow down where the change is happening.

[prestonvanloon]

Isn't there some way to get deterministic builds with bazel?
Why does some different in PATH change how protoc is compiled?

I understand this might be leading out of scope of this repository, but if you could point me in the right direction I would greatly appreciate it.

[chaoren]

Isn't there some way to get deterministic builds with bazel?

The build is deterministic, given that the PATH and its contents stays the same. Do you mean hermetic build? I agree depending on the PATH is really not hermetic.

bazel calls out to some executables on your path for binaries that it doesn't bundle (I think things like cp, mv for genrules). It's just a possibility that it might affect build output, most of the time it doesn't. bazel is just overly eager in assuming a rebuild is necessary. You can even add an empty directory to your PATH and it'll trigger a rebuild (e.g., PATH=$PATH:/bogus/bin bazel build). It really should be something the bazel team should fix.

[prestonvanloon]

Yes, hermetic builds. Thanks!

[chaoren]

There's the --action_env flag that's supposed to let you set PATH and other variables to make your build truly hermetic.

But it looks like most actions just ignore --action_env at the moment: bazelbuild/bazel#3320

[prestonvanloon]

Ah! Thanks for the pointer. I was able to resolve my original issue by adding this to my bazelrc.

build --incompatible_strict_action_env