--noremote_accept_cached cannot be used with remote downloader as of Bazel 5.0

[clint-stripe]

Description of the bug:

We use the Remote Downloader (the --experimental_remote_downloader flag) to enforce security invariants in our builds and reduce dependencies on external hosts.

We also occasionally run builds with --noremote_accept_cached.

This flag is documented as:

Whether to accept remotely cached action results.

This means we still expect to be able to use the CAS (which underlies the remote downloader service), but we do not want actions themselves to be cached.

As of Bazel 5.0 (specifically, #13769, from the git bisect I did), this fails with an error that Bazel retrieved an empty file (the e3b0c... hash), rather than the one specified in the WORKSPACE file.

Error in download_and_extract: com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException: 
Checksum was e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
but wanted bc81f1ba47ef5cc68ad32225c3d0e70b8c6f6077663835438da8d5733f917598

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

https://github.com/clint-stripe/remote-downloader-cache

We're using a different implementation of the remote cache and downloader internally, but this repo shows the same issue occurs with the open source implementation at https://github.com/buchgr/bazel-remote.

(This repro doesn't have Bazel run with --incompatible_remote_results_ignore_disk, which was added in that CL, but it does not change the behavior here either.)

Which operating system are you running Bazel on?

darwin, linux

What is the output of bazel info release?

release 5.0.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[sgowroji]

Hello @clint-stripe, Could you please provide complete steps to reproduce the above issue. Thanks!

[clint-stripe]

@sgowroji the steps to reproduce are given in the repo I linked, but I can provide them here again:

1. Clone https://github.com/clint-stripe/remote-downloader-cache
2. Run https://github.com/buchgr/bazel-remote with the configuration in that repo; you can use run-bazel-remote.sh to download and run it.
3. Use Bazel 5.0, configured to use the remote downloader and cache:

USE_BAZEL_VERSION=5.0.0 bazelisk build hello \
    --remote_cache=grpc://127.0.0.1:9092 \
    --experimental_remote_downloader=grpc://127.0.0.1:9092 \
    --disk_cache=tmp/disk_cache \
    --repository_cache=tmp/repo_cache \
    --noremote_accept_cached

This is a complete repro: the error occurs immediately when Bazel downloads the repository referenced in the WORKSPACE file.

[meisterT]

Is this related / the same as #15527?

[coeuvre]

Maybe. I will look into both.