Skip to content

Add --includTargetType and --targetType options #194

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 9, 2023
Merged

Add --includTargetType and --targetType options #194

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
da1e0ab
Write type to output as well
bz-canva Sep 22, 2023
b01243a
Add --includeTargetType and --targetType options
bz-canva Sep 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 37 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ Commands:

```terminal
Usage: bazel-diff generate-hashes [-hkvV] [--[no-]useCquery] [-b=<bazelPath>]
[--[no-]includeTargetType]
[--contentHashPath=<contentHashPath>]
[-s=<seedFilepaths>] -w=<workspacePath>
[-co=<bazelCommandOptions>]...
Expand All @@ -92,10 +93,33 @@ Usage: bazel-diff generate-hashes [-hkvV] [--[no-]useCquery] [-b=<bazelPath>]
[-so=<bazelStartupOptions>]... <outputPath>
Writes to a file the SHA256 hashes for each Bazel Target in the provided
workspace.
<outputPath> The filepath to write the resulting JSON of
dictionary target => SHA-256 values. If not
specified, the JSON will be written to STDOUT.
-b, --bazelPath=<bazelPath>
<outputPath> The filepath to write the resulting JSON to.
If not specified, the JSON will be written to STDOUT.

By default the JSON schema is a dictionary of target => SHA-256 values.
Example:
{
"//cli:bazel-diff_deploy.jar": "4ae310f8ad2bc728934e3509b6102ca658e828b9cd668f79990e95c6663f9633"
...
}

If --includeTargetType is specified, the JSON schema will include the target type (SourceFile/Rule/GeneratedFile)
Example:
{
"//cli:src/test/resources/fixture/integration-test-1.zip": "SourceFile#c259eba8539f4c14e4536c61975457c2990e090067893f4a2981e7bb5f4ef4cf",
"//external:android_gmaven_r8": "Rule#795f583449a40814c05e1cc5d833002afed8d12bce5b835579c7f139c2462d61",
"//cli:bazel-diff_deploy.jar": "GeneratedFile#4ae310f8ad2bc728934e3509b6102ca658e828b9cd668f79990e95c6663f9633",
...
}
----[no-]includeTargetType
Whether include target type in the generated JSON or not.
If false, the generate JSON schema is: {"<target>": "<sha256>"}
If true, the generate JSON schema is: {"<target>": "<type>#<sha256>"
-tt, --targetType=<targetType>
The type of targets to filter, available options are SourceFile/Rule/GeneratedFile
Only works if the JSON was generated with `--includeTargetType` enabled.
If not specified, all types of impacted targets will be returned.
-b, --bazelPath=<bazelPath>
Path to Bazel binary. If not specified, the Bazel
binary available in PATH will be used.
-co, --bazelCommandOptions=<bazelCommandOptions>
Expand Down Expand Up @@ -157,7 +181,7 @@ See https://github.com/bazelbuild/bazel/issues/17743 for more details.
### What does the SHA256 value of `generate-hashes` represent?

`generate-hashes` is a canonical SHA256 value representing all attributes and inputs into a target. These inputs
are the summation of the of the rule implementation hash, the SHA256 value
are the summation of the rule implementation hash, the SHA256 value
for every attribute of the rule and then the summation of the SHA256 value for
all `rule_inputs` using the same exact algorithm. For source_file inputs the
content of the file are converted into a SHA256 value.
Expand All @@ -167,18 +191,24 @@ content of the file are converted into a SHA256 value.
```terminal
Usage: bazel-diff get-impacted-targets [-v] -fh=<finalHashesJSONPath>
-o=<outputPath>
-tt=<targetType>
-sh=<startingHashesJSONPath>
Command-line utility to analyze the state of the bazel build graph
-fh, --finalHashes=<finalHashesJSONPath>
The path to the JSON file of target hashes for the final
revision. Run 'generate-hashes' to get this value.
-o, --output=<outputPath>
-o, --output=<outputPath>
Filepath to write the impacted Bazel targets to, newline
separated
-sh, --startingHashes=<startingHashesJSONPath>
The path to the JSON file of target hashes for the initial
revision. Run 'generate-hashes' to get this value.
-v, --verbose Display query string, missing files and elapsed time
-tt, --targetType=<targetType>
The type of targets to filter, available options are SourceFile/Rule/GeneratedFile
Only works if the JSON was generated with `--includeTargetType` enabled.
If not specified, all types of impacted targets will be returned.
-v, --verbose
Display query string, missing files and elapsed time
```

## Installing
Expand Down
20 changes: 19 additions & 1 deletion cli/src/main/kotlin/com/bazel_diff/cli/GenerateHashesCommand.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,24 @@ class GenerateHashesCommand : Callable<Int> {
)
var useCquery = false

@CommandLine.Option(
names = ["--includeTargetType"],
negatable = true,
description = ["Whether include target type in the generated JSON or not.\n"
+ "If false, the generate JSON schema is: {\"<target>\": \"<sha256>\"}\n"
+ "If true, the generate JSON schema is: {\"<target>\": \"<type>#<sha256>\" }"],
scope = CommandLine.ScopeType.INHERIT
)
var includeTargetType = false

@CommandLine.Option(
names = ["-tt", "--targetType"],
split = ",",
scope = CommandLine.ScopeType.LOCAL,
description = ["The types of targets to filter. Use comma (,) to separate multiple values, e.g. '--targetType=SourceFile,Rule,GeneratedFile'."]
)
var targetType: Set<String>? = null

@CommandLine.Option(
names = ["--cqueryCommandOptions"],
description = ["Additional space separated Bazel command options used when invoking `bazel cquery`. This flag is has no effect if `--useCquery`is false."],
Expand Down Expand Up @@ -142,7 +160,7 @@ class GenerateHashesCommand : Callable<Int> {
)
}

return when (GenerateHashesInteractor().execute(seedFilepaths, outputPath, ignoredRuleHashingAttributes)) {
return when (GenerateHashesInteractor().execute(seedFilepaths, outputPath, ignoredRuleHashingAttributes, targetType, includeTargetType)) {
true -> CommandLine.ExitCode.OK
false -> CommandLine.ExitCode.SOFTWARE
}.also { stopKoin() }
Expand Down
14 changes: 11 additions & 3 deletions cli/src/main/kotlin/com/bazel_diff/cli/GetImpactedTargetsCommand.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,14 @@ class GetImpactedTargetsCommand : Callable<Int> {
)
lateinit var finalHashesJSONPath: File

@CommandLine.Option(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this should be an array? You could desire to filter out for multiple types of target. Maybe this could also be backed by an enumeration to display the allowed values

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I have changed it to a Set to support multiple values. However, I was hesitant to change it to enum because in case of new types are added in the future, I expect the current code still work without need to add new values to the enum.

names = ["-tt", "--targetType"],
split = ",",
scope = CommandLine.ScopeType.LOCAL,
description = ["The types of targets to filter. Use comma (,) to separate multiple values, e.g. '--targetType=SourceFile,Rule,GeneratedFile'."]
)
var targetType: Set<String>? = null

@CommandLine.Option(
names = ["-o", "--output"],
scope = CommandLine.ScopeType.LOCAL,
Expand All @@ -58,8 +66,8 @@ class GetImpactedTargetsCommand : Callable<Int> {

validate()
val deserialiser = DeserialiseHashesInteractor()
val from = deserialiser.execute(startingHashesJSONPath)
val to = deserialiser.execute(finalHashesJSONPath)
val from = deserialiser.execute(startingHashesJSONPath, targetType)
val to = deserialiser.execute(finalHashesJSONPath, targetType)

val impactedTargets = CalculateImpactedTargetsInteractor().execute(from, to)

Expand All @@ -70,7 +78,7 @@ class GetImpactedTargetsCommand : Callable<Int> {
}).use { writer ->
impactedTargets.forEach {
writer.write(it)
//Should not be depend on OS
//Should not depend on OS
writer.write("\n")
}
}
Expand Down
12 changes: 6 additions & 6 deletions cli/src/main/kotlin/com/bazel_diff/hash/BuildGraphHasher.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ class BuildGraphHasher(private val bazelClient: BazelClient) : KoinComponent {
fun hashAllBazelTargetsAndSourcefiles(
seedFilepaths: Set<Path> = emptySet(),
ignoredAttrs: Set<String> = emptySet()
): Map<String, String> {
): Map<String, TargetHash> {
/**
* Bazel will lock parallel queries but this is still allowing us to hash source files while executing a parallel query
*/
Expand Down Expand Up @@ -104,7 +104,7 @@ class BuildGraphHasher(private val bazelClient: BazelClient) : KoinComponent {
sourceDigests: ConcurrentMap<String, ByteArray>,
allTargets: List<BazelTarget>,
ignoredAttrs: Set<String>
): Map<String, String> {
): Map<String, TargetHash> {
val ruleHashes: ConcurrentMap<String, ByteArray> = ConcurrentHashMap()
val targetToRule: MutableMap<String, BazelRule> = HashMap()
traverseGraph(allTargets, targetToRule)
Expand All @@ -119,13 +119,13 @@ class BuildGraphHasher(private val bazelClient: BazelClient) : KoinComponent {
seedHash,
ignoredAttrs
)
Pair(target.name, targetDigest.toHexString())
Pair(target.name, TargetHash(target.javaClass.name.substringAfterLast('$'), targetDigest.toHexString()))
}
.filter { targetEntry: Pair<String, String>? -> targetEntry != null }
.filter { targetEntry: Pair<String, TargetHash>? -> targetEntry != null }
.collect(
Collectors.toMap(
{ obj: Pair<String, String> -> obj.first },
{ obj: Pair<String, String> -> obj.second },
{ obj: Pair<String, TargetHash> -> obj.first },
{ obj: Pair<String, TargetHash> -> obj.second },
)
)
}
Expand Down
18 changes: 18 additions & 0 deletions cli/src/main/kotlin/com/bazel_diff/hash/TargetHash.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
package com.bazel_diff.hash

data class TargetHash(
val type: String, // Rule/GeneratedFile/SourceFile/...
val hash: String
) {
val hashWithType by lazy {
"${type}#${hash}"
}

fun toJson(includeTargetType: Boolean): String {
return if (includeTargetType) {
hashWithType
} else {
hash
}
}
}
17 changes: 15 additions & 2 deletions cli/src/main/kotlin/com/bazel_diff/interactor/DeserialiseHashesInteractor.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,22 @@ class DeserialiseHashesInteractor : KoinComponent {

/**
* @param file path to file that has been pre-validated
* @param targetTypes the target types to filter. If null, all targets will be returned
*/
fun execute(file: File): Map<String, String> {
fun execute(file: File, targetTypes: Set<String>? = null): Map<String, String> {
val shape = object : TypeToken<Map<String, String>>() {}.type
return gson.fromJson(FileReader(file), shape)
val result: Map<String, String> = gson.fromJson(FileReader(file), shape)
if (targetTypes == null) {
return result.mapValues { it.value.substringAfter("#") }
} else {
val prefixes = targetTypes.map { "${it}#" }.toSet()
return result.filter { entry ->
if (entry.value.contains("#")) {
prefixes.any { entry.value.startsWith(it) }
} else {
throw IllegalStateException("No type info found in ${file}, please re-generate the JSON with --includeTypeTarget!")
}
}.mapValues { it.value.substringAfter("#") }
}
}
}
16 changes: 11 additions & 5 deletions cli/src/main/kotlin/com/bazel_diff/interactor/GenerateHashesInteractor.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ class GenerateHashesInteractor : KoinComponent {
private val logger: Logger by inject()
private val gson: Gson by inject()

fun execute(seedFilepaths: File?, outputPath: File?, ignoredRuleHashingAttributes: Set<String>): Boolean {
fun execute(seedFilepaths: File?, outputPath: File?, ignoredRuleHashingAttributes: Set<String>, targetTypes:Set<String>?, includeTargetType: Boolean = false): Boolean {
return try {
val epoch = Calendar.getInstance().getTimeInMillis()
var seedFilepathsSet: Set<Path> = when {
val seedFilepathsSet: Set<Path> = when {
seedFilepaths != null -> {
BufferedReader(FileReader(seedFilepaths)).use {
it.readLines()
Expand All @@ -34,12 +34,18 @@ class GenerateHashesInteractor : KoinComponent {
val hashes = buildGraphHasher.hashAllBazelTargetsAndSourcefiles(
seedFilepathsSet,
ignoredRuleHashingAttributes
)
).let {
if(targetTypes == null) {
it
} else {
it.filter { targetTypes.contains(it.value.type) }
}
}
when (outputPath) {
null -> FileWriter(FileDescriptor.out)
else -> FileWriter(outputPath)
}.use {
it.write(gson.toJson(hashes))
}.use { fileWriter ->
fileWriter.write(gson.toJson(hashes.mapValues { it.value.toJson(includeTargetType) }))
}
val duration = Calendar.getInstance().getTimeInMillis() - epoch;
logger.i { "generate-hashes finished in $duration" }
Expand Down
28 changes: 22 additions & 6 deletions cli/src/test/kotlin/com/bazel_diff/e2e/E2ETest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,9 @@ class E2ETest {
@get:Rule
val temp: TemporaryFolder = TemporaryFolder()

@Test
fun testE2E() {
private fun CommandLine.execute(args: List<String>) = execute(*args.toTypedArray())

private fun testE2E(extraGenerateHashesArgs: List<String>, extraGetImpactedTargetsArgs: List<String>, expectedResultFile: String) {
val projectA = extractFixtureProject("/fixture/integration-test-1.zip")
val projectB = extractFixtureProject("/fixture/integration-test-2.zip")

Expand All @@ -32,24 +33,39 @@ class E2ETest {
val cli = CommandLine(BazelDiff())
//From
cli.execute(
"generate-hashes", "-w", workingDirectoryA.absolutePath, "-b", bazelPath, from.absolutePath
listOf("generate-hashes", "-w", workingDirectoryA.absolutePath, "-b", bazelPath, from.absolutePath) + extraGenerateHashesArgs
)
//To
cli.execute(
"generate-hashes", "-w", workingDirectoryB.absolutePath, "-b", bazelPath, to.absolutePath
listOf("generate-hashes", "-w", workingDirectoryB.absolutePath, "-b", bazelPath, to.absolutePath) + extraGenerateHashesArgs
)
//Impacted targets
cli.execute(
"get-impacted-targets", "-sh", from.absolutePath, "-fh", to.absolutePath, "-o", impactedTargetsOutput.absolutePath
listOf("get-impacted-targets", "-sh", from.absolutePath, "-fh", to.absolutePath, "-o", impactedTargetsOutput.absolutePath) + extraGetImpactedTargetsArgs
)

val actual: Set<String> = impactedTargetsOutput.readLines().filter { it.isNotBlank() }.toSet()
val expected: Set<String> =
javaClass.getResourceAsStream("/fixture/impacted_targets-1-2.txt").use { it.bufferedReader().readLines().filter { it.isNotBlank() }.toSet() }
javaClass.getResourceAsStream(expectedResultFile).use { it.bufferedReader().readLines().filter { it.isNotBlank() }.toSet() }

assertThat(actual).isEqualTo(expected)
}

@Test
fun testE2E() {
testE2E(emptyList(), emptyList(), "/fixture/impacted_targets-1-2.txt")
}

@Test
fun testE2EIncludingTargetType() {
testE2E(listOf("-tt", "Rule,SourceFile"), emptyList(), "/fixture/impacted_targets-1-2-rule-sourcefile.txt")
}

@Test
fun testE2EWithTargetType() {
testE2E(listOf("--includeTargetType"), listOf("-tt", "Rule,SourceFile"), "/fixture/impacted_targets-1-2-rule-sourcefile.txt")
}

@Test
fun testFineGrainedHashExternalRepo() {
// The difference between these two snapshots is simply upgrading the Guava version.
Expand Down
24 changes: 12 additions & 12 deletions cli/src/test/kotlin/com/bazel_diff/hash/BuildGraphHasherTest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,8 @@ class BuildGraphHasherTest : KoinTest {

val hash = hasher.hashAllBazelTargetsAndSourcefiles()
assertThat(hash).containsOnly(
"rule1" to "2c963f7c06bc1cead7e3b4759e1472383d4469fc3238dc42f8848190887b4775",
"rule2" to "bdc1abd0a07103cea34199a9c0d1020619136ff90fb88dcc3a8f873c811c1fe9",
"rule1" to TargetHash("Rule", "2c963f7c06bc1cead7e3b4759e1472383d4469fc3238dc42f8848190887b4775"),
"rule2" to TargetHash("Rule", "bdc1abd0a07103cea34199a9c0d1020619136ff90fb88dcc3a8f873c811c1fe9"),
)
}

Expand All @@ -86,8 +86,8 @@ class BuildGraphHasherTest : KoinTest {
whenever(bazelClientMock.queryAllSourcefileTargets()).thenReturn(emptyList())
val hash = hasher.hashAllBazelTargetsAndSourcefiles(seedFilepaths)
assertThat(hash).containsOnly(
"rule1" to "0404d80eadcc2dbfe9f0d7935086e1115344a06bd76d4e16af0dfd7b4913ee60",
"rule2" to "6fe63fa16340d18176e6d6021972c65413441b72135247179362763508ebddfe",
"rule1" to TargetHash("Rule", "0404d80eadcc2dbfe9f0d7935086e1115344a06bd76d4e16af0dfd7b4913ee60"),
"rule2" to TargetHash("Rule", "6fe63fa16340d18176e6d6021972c65413441b72135247179362763508ebddfe"),
)
}

Expand All @@ -103,10 +103,10 @@ class BuildGraphHasherTest : KoinTest {
whenever(bazelClientMock.queryAllSourcefileTargets()).thenReturn(emptyList())
val hash = hasher.hashAllBazelTargetsAndSourcefiles()
assertThat(hash).containsOnly(
"rule1" to "2c963f7c06bc1cead7e3b4759e1472383d4469fc3238dc42f8848190887b4775",
"rule2" to "bdc1abd0a07103cea34199a9c0d1020619136ff90fb88dcc3a8f873c811c1fe9",
"rule3" to "87dd050f1ca0f684f37970092ff6a02677d995718b5a05461706c0f41ffd4915",
"rule4" to "a7bc5d23cd98c4942dc879c649eb9646e38eddd773f9c7996fa0d96048cf63dc",
"rule1" to TargetHash("Rule", "2c963f7c06bc1cead7e3b4759e1472383d4469fc3238dc42f8848190887b4775"),
"rule2" to TargetHash("Rule", "bdc1abd0a07103cea34199a9c0d1020619136ff90fb88dcc3a8f873c811c1fe9"),
"rule3" to TargetHash("Rule", "87dd050f1ca0f684f37970092ff6a02677d995718b5a05461706c0f41ffd4915"),
"rule4" to TargetHash("Rule", "a7bc5d23cd98c4942dc879c649eb9646e38eddd773f9c7996fa0d96048cf63dc"),
)
}

Expand All @@ -122,10 +122,10 @@ class BuildGraphHasherTest : KoinTest {
whenever(bazelClientMock.queryAllSourcefileTargets()).thenReturn(emptyList())
val hash = hasher.hashAllBazelTargetsAndSourcefiles()
assertThat(hash).containsOnly(
"rule1" to "2c963f7c06bc1cead7e3b4759e1472383d4469fc3238dc42f8848190887b4775",
"rule2" to "bdc1abd0a07103cea34199a9c0d1020619136ff90fb88dcc3a8f873c811c1fe9",
"rule3" to "ca2f970a5a5a18730d7633cc32b48b1d94679f4ccaea56c4924e1f9913bd9cb5",
"rule4" to "bf15e616e870aaacb02493ea0b8e90c6c750c266fa26375e22b30b78954ee523",
"rule1" to TargetHash("Rule", "2c963f7c06bc1cead7e3b4759e1472383d4469fc3238dc42f8848190887b4775"),
"rule2" to TargetHash("Rule", "bdc1abd0a07103cea34199a9c0d1020619136ff90fb88dcc3a8f873c811c1fe9"),
"rule3" to TargetHash("Rule", "ca2f970a5a5a18730d7633cc32b48b1d94679f4ccaea56c4924e1f9913bd9cb5"),
"rule4" to TargetHash("Rule", "bf15e616e870aaacb02493ea0b8e90c6c750c266fa26375e22b30b78954ee523"),
)
}

Expand Down
Loading