bazel: invocations occasionally hang

[irfansharif]

Describe the problem

Noticed by @dt, where bazel run //:gazelle simply hung for over an hour. bazelbuild/bazel#11782 was recently fixed upstream, and we might want to recommend it for use internally as part of #56059.

To Reproduce

Haven't tried it, but perhaps repeatedly trying bazel invocations could trip something up. I've run into this once or twice I think.

Additional data / screenshots

~/go/src/github.com/cockroachdb/cockroach $ make bazel-generate
Running make with -j16
GOPATH set to /Users/david/go
Generating DEPS.bzl and BUILD files using gazelle
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Analyzed target //:gazelle (68 packages loaded, 7075 targets configured).
INFO: Found 1 target...
Target //:gazelle up-to-date:
  bazel-bin/gazelle-runner.bash
  bazel-bin/gazelle
INFO: Elapsed time: 37.276s, Critical Path: 6.79s
INFO: 49 processes: 13 internal, 36 darwin-sandbox.
INFO: Build completed successfully, 49 total actions
INFO: Build completed successfully, 49 total actions
INFO: Analyzed target //:gazelle (68 packages loaded, 7076 targets configured).
INFO: Found 1 target...
Target //:gazelle up-to-date:
  bazel-bin/gazelle-runner.bash
  bazel-bin/gazelle
INFO: Elapsed time: 1.920s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
^\ # hung for an hour

Proposed solution

We should try to reproduce it if possible, and verify if it's the same as bazelbuild/bazel#11782. If so, we should bump our recommended bazel version.

Epic CRDB-8036

[blathers-crl]

Hi @irfansharif, please add a C-ategory label to your issue. Check out the label system docs.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[rickystewart]

Is bazelbuild/bazel#11782 still relevant even if there's no usage of a remote cache?

[irfansharif]

I think so; reading through the discussion I walked away with the impression that there weren't appropriate timeouts set for network calls, and I assumed we're still hitting a remote cache somewhere even if not one we're hosting ourselves (I could be wrong).

[irfansharif]

Neither here nor there, but I've noticed that despite a hung command, I was able to execute other bazel commands in other terminals (bazel typically ensures that just one process is running).

[irfansharif]

Haven't seen this happen for months. Closing as stale.