Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add additional-app-test-apks support #542

Merged
merged 6 commits into from
May 1, 2019
Merged

Add additional-app-test-apks support #542

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8528c70
Parse additionalAppTestApks
bootstraponline Apr 9, 2019
74021e9
Add additional-app-test-apks support
bootstraponline Apr 10, 2019
2cb2f6c
Scope test shards to individual apks
bootstraponline Apr 29, 2019
4b522cf
Set default dispatcher when using async
bootstraponline Apr 29, 2019
672938c
Use default dispatcher when refreshing matrix
bootstraponline Apr 29, 2019
cc5be3b
Update TestRunner.kt
bootstraponline Apr 29, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,12 @@ flank:

## Local folder to store the test result. Folder is DELETED before each run to ensure only artifacts from the new run are saved.
# local-result-dir: flank

## Include additional app/test apk pairs in the run. If app is omitted, then the top level app is used for that pair.
# additional-app-test-apks:
# - app: ../test_app/apks/app-debug.apk
# test: ../test_app/apks/app-debug-androidTest.apk
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you thought of making test a list, so you don't have a bunch of usages without app?

Something like

- app: ../app.apk
  test:
    - testapk1.apk
    - testapk2.apk

- app: app2.apk
  test:
     - testfinal.apk

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that idea.

Copy link
Contributor Author

@bootstraponline bootstraponline May 1, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prototyped adding list support, in particular this complicated the code when parsing from CLI.

I think for the first version of this feature, the more verbose format is easier to maintain. The test run loop also becomes more complex as we have yet another list to iterate through.

- app: ../app.apk
  test: test1.apk
- app: ../app.apk
  test: test2.apk

I'm open to revisiting this based on feedback in the future.

# - test: ../test_app/apks/app-debug-androidTest.apk
```

### Android code coverage
Expand Down
2 changes: 1 addition & 1 deletion release_notes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## next (unreleased)

-
- [#542](https://github.com/TestArmada/flank/pull/542) Add `additional-app-test-apks` to include multiple app/test apk pairs in a single run. ([bootstraponline](https://github.com/bootstraponline))

## v6.0.1

Expand Down
7 changes: 6 additions & 1 deletion test_runner/flank.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,4 +128,9 @@ flank:

## Local folder to store the test result. Folder is DELETED before each run to ensure only artifacts from the new run are saved.
# local-result-dir: flank


## Include additional app/test apk pairs in the run. If app is omitted, then the top level app is used for that pair.
# additional-app-test-apks:
# - app: ../test_app/apks/app-debug.apk
# test: ../test_app/apks/app-debug-androidTest.apk
# - test: ../test_app/apks/app-debug-androidTest.apk
58 changes: 18 additions & 40 deletions test_runner/src/main/kotlin/ftl/args/AndroidArgs.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +1,22 @@
package ftl.args

import com.linkedin.dex.parser.DexParser
import com.linkedin.dex.parser.TestMethod
import ftl.android.AndroidCatalog
import ftl.android.IncompatibleModelVersion
import ftl.android.SupportedDeviceConfig
import ftl.android.UnsupportedModelId
import ftl.android.UnsupportedVersionId
import ftl.args.ArgsHelper.assertFileExists
import ftl.args.ArgsHelper.assertGcsFileExists
import ftl.args.ArgsHelper.calculateShards
import ftl.args.ArgsHelper.createGcsBucket
import ftl.args.ArgsHelper.createJunitBucket
import ftl.args.ArgsHelper.evaluateFilePath
import ftl.args.ArgsHelper.mergeYmlMaps
import ftl.args.ArgsHelper.yamlMapper
import ftl.args.ArgsToString.apksToString
import ftl.args.ArgsToString.devicesToString
import ftl.args.ArgsToString.listToString
import ftl.args.ArgsToString.mapToString
import ftl.args.yml.AndroidFlankYml
import ftl.args.yml.AndroidGcloudYml
import ftl.args.yml.AndroidGcloudYmlParams
import ftl.args.yml.FlankYml
Expand All @@ -26,19 +25,15 @@ import ftl.args.yml.YamlDeprecated
import ftl.cli.firebase.test.android.AndroidRunCommand
import ftl.config.Device
import ftl.config.FtlConstants
import ftl.config.FtlConstants.useMock
import ftl.filter.TestFilters
import ftl.gc.GcStorage
import ftl.util.Utils
import java.nio.file.Files
import java.nio.file.Path
import kotlinx.coroutines.runBlocking

// set default values, init properties, etc.
class AndroidArgs(
gcloudYml: GcloudYml,
androidGcloudYml: AndroidGcloudYml,
flankYml: FlankYml,
androidFlankYml: AndroidFlankYml,
override val data: String,
val cli: AndroidRunCommand? = null
) : IArgs {
Expand Down Expand Up @@ -76,21 +71,8 @@ class AndroidArgs(
override val project = cli?.project ?: flank.project
override val localResultDir = cli?.localResultDir ?: flank.localResultDir

// computed properties not specified in yaml
override val testShardChunks: List<List<String>> by lazy {
if (disableSharding) return@lazy listOf(emptyList<String>())

// Download test APK if necessary so it can be used to validate test methods
var testLocalApk = testApk
if (testApk.startsWith(FtlConstants.GCS_PREFIX)) {
runBlocking {
testLocalApk = GcStorage.downloadTestApk(this@AndroidArgs)
}
}

val filteredTests = getTestMethods(testLocalApk)
calculateShards(filteredTests, this)
}
private val androidFlank = androidFlankYml.flank
val additionalAppTestApks = cli?.additionalAppTestApks ?: androidFlank.additionalAppTestApks

init {
resultsBucket = createGcsBucket(project, cli?.resultsBucket ?: gcloud.resultsBucket)
Expand All @@ -113,21 +95,6 @@ class AndroidArgs(
devices.forEach { device -> assertDeviceSupported(device) }
}

private fun getTestMethods(testLocalApk: String): List<String> {
val allTestMethods = DexParser.findTestMethods(testLocalApk)
require(allTestMethods.isNotEmpty()) { Utils.fatalError("Test APK has no tests") }
val testFilter = TestFilters.fromTestTargets(testTargets)
val filteredTests = allTestMethods
.asSequence()
.distinct()
.filter(testFilter.shouldRun)
.map(TestMethod::testName)
.map { "class $it" }
.toList()
require(useMock || filteredTests.isNotEmpty()) { Utils.fatalError("All tests filtered out") }
return filteredTests
}

private fun assertDeviceSupported(device: Device) {
val deviceConfigTest = AndroidCatalog.supportedDeviceConfig(device.model, device.version)
when (deviceConfigTest) {
Expand Down Expand Up @@ -178,12 +145,15 @@ ${listToString(testTargetsAlwaysRun)}
disable-sharding: $disableSharding
project: $project
local-result-dir: $localResultDir
# Android Flank Yml
additional-app-test-apks:
${apksToString(additionalAppTestApks)}
""".trimIndent()
}

companion object : IArgsCompanion {
override val validArgs by lazy {
mergeYmlMaps(GcloudYml, AndroidGcloudYml, FlankYml)
mergeYmlMaps(GcloudYml, AndroidGcloudYml, FlankYml, AndroidFlankYml)
}

fun load(data: Path, cli: AndroidRunCommand? = null): AndroidArgs =
Expand All @@ -195,18 +165,26 @@ ${listToString(testTargetsAlwaysRun)}
val flankYml = yamlMapper.readValue(data, FlankYml::class.java)
val gcloudYml = yamlMapper.readValue(data, GcloudYml::class.java)
val androidGcloudYml = yamlMapper.readValue(data, AndroidGcloudYml::class.java)
val androidFlankYml = yamlMapper.readValue(data, AndroidFlankYml::class.java)

return AndroidArgs(
gcloudYml,
androidGcloudYml,
flankYml,
androidFlankYml,
data,
cli
)
}

fun default(): AndroidArgs {
return AndroidArgs(GcloudYml(), AndroidGcloudYml(AndroidGcloudYmlParams(app = ".", test = ".")), FlankYml(), "", AndroidRunCommand())
return AndroidArgs(
GcloudYml(),
AndroidGcloudYml(AndroidGcloudYmlParams(app = ".", test = ".")),
FlankYml(),
AndroidFlankYml(),
"",
AndroidRunCommand())
}
}
}
43 changes: 43 additions & 0 deletions test_runner/src/main/kotlin/ftl/args/AndroidTestShard.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
package ftl.args

import com.linkedin.dex.parser.DexParser
import com.linkedin.dex.parser.TestMethod
import ftl.config.FtlConstants
import ftl.filter.TestFilters
import ftl.gc.GcStorage
import ftl.util.Utils
import kotlinx.coroutines.runBlocking

object AndroidTestShard {

// computed properties not specified in yaml
fun getTestShardChunks(args: AndroidArgs, testApk: String): List<List<String>> {
if (args.disableSharding) return listOf(emptyList())

// Download test APK if necessary so it can be used to validate test methods
var testLocalApk = testApk
if (testApk.startsWith(FtlConstants.GCS_PREFIX)) {
runBlocking {
testLocalApk = GcStorage.download(testApk)
}
}

val filteredTests = getTestMethods(args, testLocalApk)
return ArgsHelper.calculateShards(filteredTests, args)
}

private fun getTestMethods(args: AndroidArgs, testLocalApk: String): List<String> {
val allTestMethods = DexParser.findTestMethods(testLocalApk)
require(allTestMethods.isNotEmpty()) { Utils.fatalError("Test APK has no tests") }
val testFilter = TestFilters.fromTestTargets(args.testTargets)
val filteredTests = allTestMethods
.asSequence()
.distinct()
.filter(testFilter.shouldRun)
.map(TestMethod::testName)
.map { "class $it" }
.toList()
require(FtlConstants.useMock || filteredTests.isNotEmpty()) { Utils.fatalError("All tests filtered out") }
return filteredTests
}
}
5 changes: 5 additions & 0 deletions test_runner/src/main/kotlin/ftl/args/ArgsToString.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package ftl.args

import ftl.args.yml.AppTestPair
import ftl.config.Device

object ArgsToString {
Expand All @@ -21,4 +22,8 @@ object ArgsToString {
return devices.filterNotNull()
.joinToString("\n") { "$it" }
}

fun apksToString(devices: List<AppTestPair>): String {
return devices.joinToString("\n") { (app, test) -> " - app: $app\n test: $test" }
}
}
4 changes: 1 addition & 3 deletions test_runner/src/main/kotlin/ftl/args/IArgs.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package ftl.args

import ftl.args.yml.FlankYmlParams

// Properties common to both Android and iOS
interface IArgs {
// original YAML data
val data: String
Expand All @@ -27,8 +28,5 @@ interface IArgs {
val disableSharding: Boolean
val localResultDir: String

// computed property
val testShardChunks: List<List<String>>

fun useLocalResultDir() = localResultDir != FlankYmlParams.defaultLocalResultDir
}
2 changes: 1 addition & 1 deletion test_runner/src/main/kotlin/ftl/args/IosArgs.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ class IosArgs(
val testTargets = cli?.testTargets ?: iosFlank.testTargets.filterNotNull()

// computed properties not specified in yaml
override val testShardChunks: List<List<String>> by lazy {
val testShardChunks: List<List<String>> by lazy {
if (disableSharding) return@lazy listOf(emptyList<String>())

val validTestMethods = Xctestrun.findTestNames(xctestrunFile)
Expand Down
32 changes: 32 additions & 0 deletions test_runner/src/main/kotlin/ftl/args/yml/AndroidFlankYml.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
package ftl.args.yml

import com.fasterxml.jackson.annotation.JsonIgnoreProperties
import com.fasterxml.jackson.annotation.JsonProperty

data class AppTestPair(
val app: String?,
val test: String
)

/** Flank specific parameters for Android */
@JsonIgnoreProperties(ignoreUnknown = true)
class AndroidFlankYmlParams(
@field:JsonProperty("additional-app-test-apks")
val additionalAppTestApks: List<AppTestPair> = emptyList()
) {
companion object : IYmlKeys {
override val keys = listOf("additional-app-test-apks")
}
}

@JsonIgnoreProperties(ignoreUnknown = true)
class AndroidFlankYml(
@field:JsonProperty("flank")
private val parsedFlank: AndroidFlankYmlParams? = AndroidFlankYmlParams()
) {
val flank = parsedFlank ?: AndroidFlankYmlParams()

companion object : IYmlMap {
override val map = mapOf("flank" to AndroidFlankYmlParams.keys)
}
}
2 changes: 1 addition & 1 deletion test_runner/src/main/kotlin/ftl/cli/firebase/RefreshCommand.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Reads in the matrix_ids.json file. Refreshes any incomplete matrices.
class RefreshCommand : Runnable {
override fun run() {
runBlocking {
TestRunner.refreshLastRun(AndroidArgs.default())
TestRunner.refreshLastRun(AndroidArgs.default(), emptyList())
}
}

Expand Down
Loading