[7.4.0] Compact execution log improvements #23713
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Runfiles trees are now represented with a custom `RunfilesTree` message in the compact execution log. This allows using `InputSet`s to representing all artifacts staged at canonical locations, with only symlinks and root symlinks stored flattened and with explicit runfiles paths. Since runfile paths can collide, this change makes it necessary to preserve the order of elements in an `InputSet`. The previous representation as repeated ID fields for each type (file, symlink, directory) made this impossible, so the representation has been modified to reference all direct entry IDs in a single repeated field. Since this also reduces the potential for type mismatches between the ID field type and the referenced message type, all other typed IDs are replaced with untyped ID fields. By slightly tweaking the way IDs are generated for nested entries and not emitting IDs for entries that are never referenced (e.g. `Spawn`s), IDs are now consecutive, which simplifies the (possibly concurrent) bookkeeping for consumers by allowing them to use an array to store the entries. Progress on bazelbuild#18643. RELNOTES: The compact execution log now stores runfiles in a more compact representation that should reduce the memory overhead and log output size, in particular for test spawns. This change required breaking changes to the (experimental) log format. Closes bazelbuild#23321. PiperOrigin-RevId: 676773599 Change-Id: I010653681ffa44557142bf25009e9178b5d68515 (cherry picked from commit c2f539c)
github-actions
bot
added
awaiting-review
tjgq
approved these changes
Sep 30, 2024
tjgq
added
awaiting-PR-merge
This was referenced Sep 30, 2024
iancha1992
removed
the
awaiting-PR-merge
Merged
via the queue into
bazelbuild:release-7.4.0
with commit Sep 30, 2024
53b04fe
51 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry-picks the following changes:
spawn.proto(Fix naming inconsistency inspawn.proto#23706)The cherry-picks required introducing a
Map<Artifact, RunfilesTree>shim toRunfilesSupplierthat matches the Bazel 8 way of obtaining aRunfilesTreefrom a runfiles middleman viaInputMetadataProvider.Closes #23683
Closes #23710
Closes #23711
Closes #23734