Releases   |   Documentation   

Gradle plugin wrapper for Marathon test runner

TL;DR

Marathon is a fast, platform-independent test runner focused on performance and stability.

Marathon implements multiple key concepts of test execution such as test batching, device pools, test sharding, test sorting, preventive retries as well as post-factum retries. By default, most of these are set to conservative defaults but custom configurations are encouraged for those who want to optimize performance and/or stability.

Marathon's primary focus is on full control over the balance between stability of test execution, testing performance and cost.

For more information see the documentation

Overview

Tradeoffs using Gradle Plugin

Pros	Cons
Configuration using Gradle syntax	Requires project sync before testing starts
No installation of marathon CLI required	Less flexibility in choosing AGP+Gradle versions. CLI is independent of your Gradle setup
	Harder to manage when you have more than 1 test run configuration
	Missing features, e.g. multi-module testing

Setup

Marathon gradle plugin is published to plugins.gradle.org. To apply the plugin:

Gradle KTS

plugins {
  id("com.malinskiy.marathon") version "X.X.X"
}

Gradle

plugins {
  id 'com.malinskiy.marathon' version 'X.X.X'
}

Configuration

Configuration for Gradle Plugin can only be done via Gradle, i.e. you can't use Marathonfile as configuration when running tests using Gradle Plugin.

Here is an example of gradle config using Kotlin DSL (for more information about the parameters see the documentation):

marathon {
  name = "sample-app tests"
  baseOutputDir = "./marathon"
  outputConfiguration {
    maxPath = 1024
  }
  analytics {
    influx {
      url = "http://influx.svc.cluster.local:8086"
      user = "root"
      password = "root"
      dbName = "marathon"
    }
  }
  poolingStrategy {
    operatingSystem = true
  }
  shardingStrategy {
    countSharding {
      count = 5
    }
  }
  sortingStrategy {
    executionTime {
      percentile = 90.0
      executionTime = Instant.now().minus(3, ChronoUnit.DAYS)
    }
  }
  batchingStrategy {
    fixedSize {
      size = 10
    }
  }
  flakinessStrategy {
    probabilityBased {
      minSuccessRate = 0.8
      maxCount = 3
      timeLimit = Instant.now().minus(30, ChronoUnit.DAYS)
    }
  }
  retryStrategy {
    fixedQuota {
      totalAllowedRetryQuota = 200
      retryPerTestQuota = 3
    }
  }
  filteringConfiguration {
    allowlist {
      add(SimpleClassnameFilterConfiguration(".*".toRegex()))
    }
    blocklist {
      add(SimpleClassnameFilterConfiguration("$^".toRegex()))
    }
  }
  includeSerialRegexes = emptyList()
  excludeSerialRegexes = emptyList()
  uncompletedTestRetryQuota = 100
  ignoreFailures = false
  isCodeCoverageEnabled = false
  fallbackToScreenshots = false
  testOutputTimeoutMillis = 30_000
  debug = true
  autoGrantPermission = true
}

Execute

Executing your tests via gradle is done via calling generated marathon gradle task, for example marathonDebugAndroidTest. All the test tasks will start with marathon prefix, for example marathonDebugAndroidTest. These tasks will be created for all testing flavors including multi-dimension setup.

$ gradle :app:marathonDebugAndroidTest

Contributing

See contributing docs

License

Marathon codebase is GPL 2.0 LICENSE