Bazelisk is pulling a non existing release ( 3.5.1 ) when invoked w/o .bazelversion

[jerrymarino]

Invoking bazelisk build is attempting to pull a non-existent release.

This afternoon, a handful of xchammer github jobs started failing that used bazelisk. For example:
https://github.com/pinterest/xchammer/pull/259/checks?check_run_id=1168118720

The xchammer CI runs bazelisk and we don't have a version specified in the repository. Without the version, it attempts to pull a non-existing URL:

2020/09/25 21:40:42 Downloading https://releases.bazel.build/3.5.1/release/bazel-3.5.1-darwin-x86_64...
2020/09/25 21:40:42 could not download Bazel: HTTP GET https://releases.bazel.build/3.5.1/release/bazel-3.5.1-darwin-x86_64 failed with error 404
make: *** [build-impl] Error 1
Error: Process completed with exit code 2.

I checked on Bazel releases this afternoon, and it looks like there is no 3.5.1 bazel release: https://github.com/bazelbuild/bazel/releases

[philwo]

@fweikert Any idea what causes this? Is this related to your work where we fetch the Version info from GCS instead of GitHub?

Maybe the code doesn’t recognize that 3.5.1 isn’t a released version yet and only has an rc1?

[philwo]

@jerrymarino I’m so sorry for this :( I’m not sure yet what causes this, whether it’s an issue with Bazelisk or something weird about the Bazel 3.5.1rc1 release, but will look into it tomorrow.

[junyer]

RE2 is also seeing this for macOS builds with GitHub Actions. We weren't actually intending to use Bazelisk, so I'm guessing that the GitHub-hosted runners have bazel pointing to Bazelisk? Is there a flag or an environment variable that can be passed that will prevent Bazelisk from doing anything? As opposed to configuring a version, I mean.

P.S. Hi, @philwo!

[philwo]

I found the reason for this. Bazelisk has two methods to find out what the latest Bazel is:

1. Query the available Bazel releases in our repo via the GitHub API
2. Infer it from the versions available on our Google Cloud Storage release download bucket

Bazelisk uses GitHub API by default, but falls back to GCS, if that doens't work for some reason. Unfortunately the GCS strategy has a bug - it sees a 3.5.1 folder on GCS and assumes that's the latest version, without verifying that this is already a final release. But as the 3.5.1 folder only contains the rc1 version, the download then fails.

While we fix this, you have two workarounds for this:

1. Add a .bazelversion file to your repo, then Bazelisk will use that version and not query for the latest one. This is a good idea in general, because you can then test Bazel version upgrades simply by bumping the version in that file and upgrade at your own pace, instead of having your build break when Bazel releases an incompatible version.
2. Alternatively, you'd have to ensure that Bazelisk can successfully query GitHub's API for the version information. This usually works fine, but on GitHub Actions you'll often run into rate-limiting (HTTP error 403 for the API request) as the CI workers all share common IPs. To fix this, you'll have to pass GitHub Action's API Token to Bazelisk via the environment variable BAZELISK_GITHUB_TOKEN.

Reference for how that works:

• About the token: https://docs.github.com/en/free-pro-team@latest/actions/reference/authentication-in-a-workflow#using-the-github_token-in-a-workflow
• How to pass the token into an env var: https://docs.github.com/en/free-pro-team@latest/actions/reference/encrypted-secrets#using-encrypted-secrets-in-a-workflow

How I think this should work (for re2): google/re2#282

Sorry for the problems 😞

[junyer]

Confirmed that setting BAZELISK_GITHUB_TOKEN mitigates the problem. Many thanks, @philwo! :D

[philwo]

@junyer Yay! Thanks for reporting back (and nice to see you around here 😊)

[fweikert]

This is what happened:

Bazelisk saw that the latest version was 3.6.0, but without a release -> it returned the second most recent version, 3.5.1, without actually checking whether that version had a release either. Since there was no 3.5.1 release you got a 404. I'll send a fix.