Is Remote Build without the Bytes handled correctly when using the S3 proxy backend with a lifecycle policy?

[WSUFan]

Hi guys,

I have a question.
For example, I setup a 7-day life policy on a S3 bucket. Suppose the bazel remote cache cannot find ac/cas locally and it asks S3. In the meantime, the S3 life policy triggers and it deletes corresponding ac/cas items. After that, the bazel client seems to throw digest not found exeception and fails the build. So it seems that the bazel remote cache will not make a request to validate the ac/cas items in S3?

Thanks

[mostynb]

Hi.

I think that's correct (if unfortunate).

IIRC in bazel's current setup, the build-without-the-bytes feature requires that none of the referenced (but not downloaded) blobs are deleted during the build. This isn't great, but I think people are working to make bazel cope with this scenario without crashing- see bazelbuild/bazel#14126 and bazelbuild/bazel#16660. You might want to wait for this feature to land before using build-without-the-bytes and cache eviction in the proxy backend.

This could be improved on the bazel-remote side if we could figure out a way to implement a performant LRU-like eviction policy for the proxy backends. https://github.com/znly/bazel-cache has something that apparently works for GCS, but I don't have a GCS account to experiment with so I haven't tried implementing that in bazel-remote, and I haven't had time to investigate a similar solution for s3.

[WSUFan]

Thanks for the reply.

From this closed issue #185, bazel-remote can handle Build without the Bytes correctly using the local disk cache. So it seems like bazel-remote is able to know all the referenced blobs for each AC, right? Based on that, maybe we could make a request to S3 and ask it to update the timestamp (just like the option --s3.update_timestamps does)?

[mostynb]

Hmmm. I see that the s3.update_timestamps feature was only used when fetching blobs from the backend, but not when checking for their existence. If we also update blob timestamps after they're confirmed to exist (ie in Contains()), then they would be kept alive longer. You might like to try a build from #625 to see if that helps.

I don't think this will be sufficient however, because if bazel-remote finds a cache hit in the disk cache, it returns immediately without talking to the proxy backend. If we added such a call synchronously then I think that would slow bazel-remote down quite a bit, and if we added an async call we would need to be careful not to spike the number of active goroutines.

[WSUFan]

Thanks! I think this is enough for my case. I will close this issue. Thanks for the help!