When using http_proxy, can upload an empty action.

[uri-canva]

This results in errors similar to:

ERROR: $WORKSPACE/$PACKAGE/BUILD:1:1: output '$PACKAGE/lib$TARGET-class.jar' was not created
ERROR: $WORKSPACE/$PACKAGE/BUILD:1:1: output '$PACKAGE/lib$TARGET.jar_manifest_proto' was not created
ERROR: $WORKSPACE/$PACKAGE/BUILD:1:1: output '$PACKAGE/lib$TARGET.jdeps' was not created
ERROR: $WORKSPACE/$PACKAGE/BUILD:1:1: output '$PACKAGE/lib$TARGET-native-header.jar' was not created
ERROR: $WORKSPACE/$PACKAGE/BUILD:1:1: not all outputs were created or valid

when using java rules for example.

[uri-canva]

This started happening after we started using http_proxy, it never happened before that, so I strongly suspect that codepath. Also the issue ends up requiring deleting both the cache we proxy from and the cache we proxy to, which points me to the upload path rather than the download path.

[uri-canva]

It's not clear to me when the size of the artefact being uploaded could be 0 in 700d941#diff-b759c035e06bbab37422eac5ba2473b6R41.

[uri-canva]

Also not sure an empty artefact would be a zero sized one, as opposed to an empty zip stream or something, I did not save the cache directories when this error happened. Will do it if it happens again.

[uri-canva]

Narrowed it down: it's not an empty artefact, it's an empty action.

[uri-canva]

The action is a zero byte file.

[ob]

It's not clear to me when the size of the artefact being uploaded could be 0 in 700d941#diff-b759c035e06bbab37422eac5ba2473b6R41.

Bazel can cache empty artifacts[1] (e.g. when one of the inputs is /dev/null). I don't know about empty actions though...

[1] See bazelbuild/bazel#6260 for example

[nicolov]

I added that check because I kept seeing errors in cached builds, and traced it down to bazel uploading empty files. For example, I was seeing linker crashes as the .o files were (unexpectedly) empty. This was with bazel 0.54, so maybe it can safely be removed now.

[uri-canva]

I think empty artifacts are ok, especially if they're 0 byte files as opposed to no files at all. They are a valid output after all. Empty actions seem to be the invalid case.

[nicolov]

I agree in principle that empty outputs should be ok. However, over tens of thousands of builds with 0.5.4, I've seen bazel mistakenly upload empty files. These errors would show up as linker errors (if the truncated file was an object file), or even compiler errors (if it was a generated header). Any builds that touched the truncated file would fail until the cache was wiped.

Many things have changed in bazel since then, so the bug might very well have gone away.

[mostynb]

I added some ActionResult validation a while back which might help catch corrupt action cache files, if they are corrupted by bazel-remote itself. On upload we now make sure that the worker is set in execution_metadata, which makes the serialized data non-empty. Then we check that ActionCache items aren't empty before returning them to the client.

However technically speaking a zero-length ActionResult proto is valid, so we can't reject a zero-sized ActionResult if that's what the client or REAPI worker uploads.

[mostynb]

I think we've done all we can here, closing.