-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
review of practices #30
base: main
Are you sure you want to change the base?
Changes from 1 commit
6e0b591
f94a984
166d8d7
1fb3bf5
88d0741
497d515
2f24e61
1815a54
3e9b9aa
eab7416
c7f7570
8124fa0
3398f94
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,64 +1,67 @@ | ||
# Test runners | ||
|
||
Test runner is responsible for tests run and providing test result for us. | ||
Test runner is a test component responsible for. | ||
1. Preparing the test runs | ||
2. Providing test results | ||
|
||
`AndroidJunitRunner` — Official solution and low-level instrument. It requires a lot of effort from engineers to run | ||
tests on CI and make them stable. | ||
`AndroidJunitRunner` — The official solution and low-level instrument. It requires a lot of effort from engineers to run | ||
tests on the CI and make them stable. | ||
|
||
It's worth to mention — tools are getting better year by year. However, some basic functionality still doesn't work from | ||
the box properly. | ||
It's worth mentioning that UI testing tools are getting better year by year. However, some basic functionality still doesn't work out | ||
of the box properly. | ||
|
||
### 1. Problems with AndroidJunitRunner: | ||
|
||
* Overcomplicated solution with clearing | ||
* Overcomplicated solution with state clearing | ||
<br> | ||
_It would be good to have only one flag which does application clearing for us. | ||
_It would be good to do the state clearing of the application by setting a flag. | ||
<br> | ||
It exists, however to have scalability on CI and opportunity to use filters, you still have to | ||
install `test-services.apk` and `orchestrator.apk` to each device manually_ | ||
It exists, however to make it scalable on the CI as well as the opportunity to use filters, you still have to | ||
install `test-services.apk` and `orchestrator.apk` on each device manually_ | ||
|
||
|
||
* Impossibility to scale | ||
<br> | ||
_As soon as you started your tests, it's impossible to connect more devices to tests run on fly_ | ||
_As soon as you started your tests, it's impossible to let more devices join the test execution on the fly_ | ||
|
||
* Impossibility to prevent flakiness | ||
<br> | ||
_Flakiness is one of the main problems in instrumented testing. Test runner should play role as a latest flakiness | ||
protection level, like support retries from the box or other strategies_ | ||
_Flakiness is one of the main problems in instrumented testing. Test runners should provide some mechanisms to fight against | ||
flakiness, like support to retry failing tests among other strategies_ | ||
|
||
* Impossibility to validate flakiness | ||
<br> | ||
_Flakiness can be validated by running each test multiple times, and if test pass N/N, it's not flaky. It would be | ||
_Flakiness can be validated by running each test multiple times. If the test passes every single time it runs, it's not flaky. It would be | ||
great to launch each test 100 times by one command_ | ||
|
||
* Poor test report | ||
<br> | ||
_Default test report doesn't show any useful information. As an engineer, I want to see a video of the test, logs and | ||
to make sure that test hasn't been retried. Otherwise, I'd like to see retries and each retry video and logs._ | ||
_The default test report doesn't show enough valuable information for each test. As an engineer, I want to see a video of the test and its logs. Moreover, I'd like to know whether | ||
the test has been retried. If yes, I'd also like to see how many retries, and their corresponding videos and logs_ | ||
|
||
* Impossibility to retry | ||
<br> | ||
_It's possible to do only via special test rule which does retry for us. However, it's up to test runner to retry each | ||
test, as instrumented process can be crashed and device less reliable than host machine._ | ||
_It's only possible to do it via a special test rule which does the retries for us. However, it's up to the test runner to retry each | ||
test. That's because: instrumented process might have crashed and device less reliable than host machine._ | ||
1. *The instrumented process might have crashed*. In this case the test rule may not execute the code where the retry actually happens. | ||
2. ** | ||
<br> | ||
_Also, it should be possible to define maximum retry count: Imagine, your application crashed on start. We shouldn't | ||
retry each test in that case because there is no sense to overload build agents on CI._ | ||
_Also, it should be possible to define maximum retry count: Imagine, your application reliably crashes on start, and you have plenty of tests executing that code. We shouldn't | ||
retry each test in that case: we would overload build agents on the CI with tests that are doomed to fail._ | ||
|
||
|
||
* Impossibility to record a video | ||
<br> | ||
_It's possible to achieve and implement manually, however It would be really great to have such functionality from the | ||
box_ | ||
_It's possible to achieve and implement manually. However, it would be really great to have such a functionality already built-in_ | ||
|
||
Almost all of that problems possible to solve, but it can take weeks or even months of your time. Beside running tests, | ||
Almost all of those problems can be solved, but it can take weeks or even months of your time. Beside running tests, | ||
you also need to care about writing tests which is challenging as well. | ||
<br> | ||
It would be great to have that problems solved from the box | ||
Having those problems solved for you lets you focus on other tasks. | ||
|
||
### 2. Open source test runners | ||
|
||
All of them used `AndroidJunitRunner` under the hood, as it's the only possibility tun run instrumented tests. | ||
All of them use `AndroidJunitRunner` under the hood, as it's the only possibility to run instrumented tests. | ||
|
||
#### [:green_square: 2.1 Marathon](https://github.com/MarathonLabs/marathon) | ||
|
||
|
@@ -72,25 +75,25 @@ do the whole job for you. | |
➕ Dynamic test batching (test count/test duration) <br> | ||
➕ Smart retries with a quotas <br> | ||
➕ Screenshots & video out of the box <br> | ||
➕ Improved test report with video & logs <br> | ||
➕ Automatically rebalanced test execution if connecting/disconnecting devices on fly <br> | ||
➕ Improved test report with video & logs <br> | ||
➕ Automatically rebalanced test execution if connecting/disconnecting devices on the fly <br> | ||
➕ Pull files from the device after test run, | ||
e.g. [allure-kotlin](https://github.com/allure-framework/allure-kotlin) <br> | ||
➕ Basic [Allure](https://github.com/allure-framework) support out of the box <br> | ||
➕ Basic [Allure](https://github.com/allure-framework) support out of the box <br> | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is not clear what Allure does for someone who has not used it: |
||
➕ adb client `ddmlib` replacement: [Adam](https://github.com/Malinskiy/adam) <br> | ||
➕ Cross-platform (iOS support) <br> | ||
➕ Fragmented test execution (similar to AOSP's sharding): split large testing suites into multiple CI builds <br> | ||
➕ Parallel execution of parameterised tests <br> | ||
➕ Interactions with adb/emulator from within a test (e.g. fake fingerprint or GPS) <br> | ||
➕ Interactions with adb/emulator from within a test (e.g. fake fingerprint or GPS) <br> | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Looks like Kaspresso does already half of these things :P |
||
➕ Code coverage support <br> | ||
➕ Testing multi-module projects in one test run <br> | ||
➕ Testing multi-module projects in one test run <br> | ||
➕ Flakiness fixing mode to verify test passing probability improvements <br> | ||
|
||
➖ Doesn't auto-scale devices <br> | ||
_(Marathon will utilise more devices in runtime if some other system connects more to the adb, but marathon itself will | ||
not spawn more emulators for you)_<br> | ||
➖ HTML report doesn't contain test retry information (but the Allure report does) <br> | ||
➖ For complex test executions that solve test flakiness requires an installation of TSDB (InfluxDB or Graphite) <br> | ||
➖ For complex test executions that solve test flakiness requires to install TSDB (InfluxDB or Graphite) <br> | ||
|
||
[Documentation](https://marathonlabs.github.io/marathon/) | ||
|
||
|
@@ -100,15 +103,15 @@ Powerful test runner. Works directly with `Kubernetes` | |
|
||
➕ Easy data clearing _(without an Orchestrator)_ <br> | ||
➕ Auto-scaling on fly _(There is a coroutine in the background which tries to connect more devices)_ | ||
➕ Retries | ||
➕ Retries | ||
|
||
➖ Complicated adoption <br> | ||
|
||
This test runner has been using by Avito company for 4+ years and runs thousands tests every day. It's not as powerful | ||
as Marathon, however it doesn't have an analogue in terms of auto scaling from the box.<br> | ||
This test runner has been used by Avito for 4+ years and runs thousands of tests every day. It's not as powerful | ||
as Marathon, however it doesn't have an analogue in terms of auto-scaling out of the box.<br> | ||
If you want to run your UI tests on pull requests in a large teams, this test runner is one of the best option. | ||
|
||
Engineers from Avito are ready to help with adoption. You can contact to [Dmitriy Voronin](https://github.com/dsvoronin) | ||
Engineers from Avito are ready to help with adoption. You can reach out to [Dmitriy Voronin](https://github.com/dsvoronin) | ||
|
||
[Documentation](https://avito-tech.github.io/avito-android/test_runner/TestRunner/) | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think explaining why it can help to add test on the fly and what is the case would help.
I was thinking about running test on a device cloud? There could be more available after a while? @NoNews what is the case exactly?