Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[5.1] Remote: Action should not be successful and cached if outputs were not created #15071

Merged
merged 3 commits into from
Mar 18, 2022
Merged

[5.1] Remote: Action should not be successful and cached if outputs were not created #15071

merged 3 commits into from
Mar 18, 2022

Commits on Mar 18, 2022

  1. Add a method, Spawn#isOutputMandatory, to indicate whether a Spawn mu… 

    …st create an output by the end of its execution. If not, a Spawn (like a test) may succeed without necessarily creating that output.

The 4 categories of actions that do this are:

1. Tests (tests can create XML and other files, but may not).
2. Java compilations with reduced classpaths (the initial compilation with a reduced classpath may fail, but produce a usable jdeps proto file, so the Spawn will claim that it succeeded so that the action can process the proto file).
3. Extra actions with a dummy output that is produced locally, not by the Spawn. However, by changing the outputs of the Spawn to be empty, this situation is avoided.
4. C++ compilations with coverage (a .gcno coverage file is not produced for an empty translation unit).

In particular, all SpawnActions' Spawns have #isMandatoryOutput always return true, and there should be no new cases of #isMandatoryOutput returning false besides the ones added in this CL.

PiperOrigin-RevId: 425616085
    @janakdr @coeuvre
    janakdr authored and coeuvre committed Mar 18, 2022
    Configuration menu
    Copy the full SHA
    ac3439c View commit details
    Browse the repository at this point in the history

  2. Remote: Don't upload action result if declared outputs are not created. 

    Even if the exit code is 0. Missing declared outputs will be detected by Bazel later and fail the build, so avoid uploading this false positive cache entry.

Wrap buildUploadManifest inside a `Single.fromCallable` since there are many IOs (both the check we add in this PR and stats already there). If `--experimental_remote_cache_async` is set, these IOs will now be executed in a background thread.

Fixes bazelbuild#14543.

Closes bazelbuild#15016.

PiperOrigin-RevId: 434448255
    @coeuvre
    coeuvre committed Mar 18, 2022
    Configuration menu
    Copy the full SHA
    f32814d View commit details
    Browse the repository at this point in the history

  3. Remote: Check declared outputs when downloading outputs. 

    An AC entry that misses declared outputs of an action is invalid because, if Bazel accepts this from remote cache, it will detect the missing declared outputs error at a later stage and fail the build.

This PR adds a check for mandatory outputs when downloading outputs from remote cache. If a mandatory output is missing from AC entry, Bazel will ignore the cache entry so build can continue.

Also fixes an issue introduced by bazelbuild#15016 that tests result are not uploaded to remote cache. Test spawns declare all the possible outputs but they usually don't generate them all. This change fixes that by only check for mandatory outputs via `spawn.isMandatoryOutput()`.

Fixes bazelbuild#14543.

Closes bazelbuild#15051.

PiperOrigin-RevId: 435307260
    @coeuvre
    coeuvre committed Mar 18, 2022
    Configuration menu
    Copy the full SHA
    722ee13 View commit details
    Browse the repository at this point in the history