Use package:patrol for Android e2e tests. (#1137)

Use [`package:patrol`](https://pub.dev/packages/patrol) for integration/e2e tests for Android. The biggest benefit is that each test can run from a completely clean state, which is not possible with the `integration_test` package. Because of this we shouldn't have the issues with e.g. the auth state being kept for each test (or logging out and back in which breaks stuff). For other platforms I kept our old tests (renamed to `integration_test/integration_test_old.dart`): * Web should theoretically work as long as no native interaction is used, but I couldn't set it up practically. There is an [open issue for web ](leancodepl/patrol#733) in their repo. * iOS should work, but I couldn't get it running. * macOS is never mentioned, so I don't know if it's possible to run tests for mac. From https://patrol.leancode.co/: > Patrol lets you [access native features of the platform](https://patrol.leancode.co/native/overview) that the Flutter app is running on. Finally, you can interact with permission dialogs, notifications, WebViews, change device settings, toggle Wi-Fi, and much more – and you can code this very easily in plain Dart. > > Patrol also provides a [new custom finder system](https://patrol.leancode.co/finders/overview) that extends Flutter's default finders, making them shorter and easier to understand. Patrol's custom finders, coupled with [Hot Restart](https://patrol.leancode.co/cli-commands/develop), make writing integration tests dramatically faster, easier and more fun! Command to running the tests: ``` patrol build android \ --flavor prod \ --dart-define USER_1_EMAIL=example@sharezone.net \ --dart-define USER_1_PASSWORD=foobar \ -t integration_test/app_test.dart ```
SharezoneApp · Nov 6, 2023 · 210a8b6 · 210a8b6
1 parent b441d82
commit 210a8b6
Show file tree

Hide file tree

Showing 18 changed files with 409 additions and 193 deletions.
diff --git a/.github/workflows/integration_tests_app_ci.yml b/.github/workflows/integration_tests_app_ci.yml
@@ -79,7 +79,7 @@ jobs:
     # (~5% of the time) build takes longer and then is a long timeout needed.
     defaults:
       run:
-        working-directory: app/android
+        working-directory: app
     timeout-minutes: 90
     steps:
       - uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608
@@ -99,18 +99,10 @@ jobs:
           flutter-version: ${{ env.FLUTTER_VERSION }}
           channel: ${{ env.FLUTTER_CHANNEL }}
 
-      - name: Build Instrumentation Test
-        env:
-          # In gradlew, we can't just pass --dart-define as we do with "flutter
-          # test". We need to convert the key and the value into base64, like
-          # "USER_1_EMAIL=user@example.com" becomes to
-          # "VVNFUl8xX0VNQUlMPXVzZXJAZXhhbXBsZS5jb20=".
-          #
-          # The secrets already contain the base64 encoded values. We don't
-          # encode them in the workflow file, because we could easily leak the
-          # base64 encoded version in logs.
-          USER_1_EMAIL_BASE64: ${{ secrets.INTEGRATION_TEST_USER_1_EMAIL_DART_DEFINE_BASE64 }}
-          USER_1_PASSWORD_BASE64: ${{ secrets.INTEGRATION_TEST_USER_1_PASSWORD_DART_DEFINE_BASE64 }}
+      - name: Install patrol cli
+        run: flutter pub global activate patrol_cli ^2.2.1
+
+      - name: Run Flutter build
         run: |
           # Flutter build is required to generate files in android/ to build the
           # gradle project.
@@ -119,14 +111,17 @@ jobs:
           # Firebase Test Lab with the dev flavor. We always got "No tests
           # found.".
           flutter build apk \
-            --target=lib/main_prod.dart \
-            --flavor prod \
-            --config-only
+          --target=lib/main_prod.dart \
+          --flavor prod \
+          --config-only
 
-          ./gradlew app:assembleProdDebugAndroidTest
-          ./gradlew app:assembleProdDebug \
-            -Ptarget=integration_test/app_test.dart \
-            -Pdart-defines="$USER_1_EMAIL_BASE64,$USER_1_PASSWORD_BASE64"
+      - name: Build Instrumentation Test
+        run: |
+          patrol build android \
+          --flavor prod \
+          --dart-define USER_1_EMAIL=${{ secrets.INTEGRATION_TEST_USER_1_EMAIL }} \
+          --dart-define USER_1_PASSWORD=${{ secrets.INTEGRATION_TEST_USER_1_PASSWORD }} \
+          -t integration_test/app_test.dart
 
       - name: Setup credentials
         env:
@@ -153,10 +148,12 @@ jobs:
         run: |
           gcloud firebase test android run \
             --type instrumentation \
-            --app ../build/app/outputs/apk/prod/debug/app-prod-debug.apk \
-            --test ../build/app/outputs/apk/androidTest/prod/debug/app-prod-debug-androidTest.apk \
+            --app build/app/outputs/apk/prod/debug/app-prod-debug.apk \
+            --test build/app/outputs/apk/androidTest/prod/debug/app-prod-debug-androidTest.apk \
             --device model=Pixel2,version=30,locale=en,orientation=portrait \
-            --timeout 3m
+            --timeout 10m \
+            --use-orchestrator \
+            --environment-variables clearPackageData=true
 
       # It can easily happen that a dependency changed but the .lock file is not
       # updated. Or other cases where files are changed during a build.
@@ -229,7 +226,7 @@ jobs:
           # https://github.com/flutter/flutter/issues/88690
           fvm flutter drive \
             --driver=test_driver/integration_test.dart \
-            --target=integration_test/app_test.dart \
+            --target=integration_test/integration_test_old.dart \
             --flavor prod \
             --dart-define=USER_1_EMAIL=$USER_1_EMAIL \
             --dart-define=USER_1_PASSWORD=$USER_1_PASSWORD \
@@ -274,7 +271,7 @@ jobs:
           chromedriver --port=4444 &
           fvm flutter drive \
             --driver=test_driver/integration_test.dart \
-            --target=integration_test/app_test.dart \
+            --target=integration_test/integration_test_old.dart \
             --flavor dev \
             --dart-define=USER_1_EMAIL=$USER_1_EMAIL \
             --dart-define=USER_1_PASSWORD=$USER_1_PASSWORD \

diff --git a/app/android/app/build.gradle b/app/android/app/build.gradle
@@ -56,15 +56,22 @@ android {
         disable 'InvalidPackage'
     }
 
+    // From https://patrol.leancode.co/getting-started#create-a-simple-integration-test
+    testOptions {
+        execution "ANDROIDX_TEST_ORCHESTRATOR"
+    }
+
+
     defaultConfig {
         applicationId "de.codingbrain.sharezone"
         minSdkVersion 21
         targetSdkVersion 33
         versionCode flutterVersionCode.toInteger()
         versionName flutterVersionName
         multiDexEnabled true
-        // From https://pub.dev/packages/integration_test
-        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
+        // From https://patrol.leancode.co/getting-started#create-a-simple-integration-test
+        testInstrumentationRunner "pl.leancode.patrol.PatrolJUnitRunner"
+        testInstrumentationRunnerArguments clearPackageData: "true"
     }
 
     signingConfigs {
@@ -114,6 +121,8 @@ dependencies {
     testImplementation 'junit:junit:4.12'
     androidTestImplementation 'androidx.test:runner:1.2.0'
     androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
+    // From https://patrol.leancode.co/getting-started#create-a-simple-integration-test
+    androidTestUtil "androidx.test:orchestrator:1.4.2"
 }
 
 apply plugin: 'com.google.firebase.firebase-perf'

diff --git a/app/android/app/src/androidTest/java/de/codingbrain/sharezone/MainActivityTest.java b/app/android/app/src/androidTest/java/de/codingbrain/sharezone/MainActivityTest.java
@@ -10,13 +10,32 @@
 
 package de.codingbrain.sharezone;
 
-import androidx.test.rule.ActivityTestRule;
-import dev.flutter.plugins.integration_test.FlutterTestRunner;
-import org.junit.Rule;
+import androidx.test.platform.app.InstrumentationRegistry;
+import org.junit.Test;
 import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
+import pl.leancode.patrol.PatrolJUnitRunner;
 
-@RunWith(FlutterTestRunner.class)
+@RunWith(Parameterized.class)
 public class MainActivityTest {
-  @Rule
-  public ActivityTestRule<MainActivity> rule = new ActivityTestRule<>(MainActivity.class, true, false);
+    @Parameters(name = "{0}")
+    public static Object[] testCases() {
+        PatrolJUnitRunner instrumentation = (PatrolJUnitRunner) InstrumentationRegistry.getInstrumentation();
+        instrumentation.setUp(MainActivity.class);
+        instrumentation.waitForPatrolAppService();
+        return instrumentation.listDartTests();
+    }
+
+    public MainActivityTest(String dartTestName) {
+        this.dartTestName = dartTestName;
+    }
+
+    private final String dartTestName;
+
+    @Test
+    public void runDartTest() {
+        PatrolJUnitRunner instrumentation = (PatrolJUnitRunner) InstrumentationRegistry.getInstrumentation();
+        instrumentation.runDartTest(dartTestName);
+    }
 }
diff --git a/app/integration_test/.gitignore b/app/integration_test/.gitignore
@@ -0,0 +1 @@
+test_bundle.dart
diff --git a/app/integration_test/README.md b/app/integration_test/README.md
@@ -2,7 +2,10 @@
 
 Unit tests and widget tests are handy for testing individual classes, functions, or widgets. However, they generally don’t test how individual pieces work together as a whole, or capture the performance of an application running on a real device. These tasks are performed with integration tests.
 
-Integration tests are written using the [integration_test](https://github.com/flutter/flutter/tree/master/packages/integration_test) package, provided by the SDK.
+Integration tests for Android are written using the [patrol ](https://pub.dev/packages/patrol) package.
+
+Integration tests for all other platforms are written using the [integration_test](https://github.com/flutter/flutter/tree/master/packages/integration_test) package, provided by the SDK.
+
 
 ## How to run integration tests
 
@@ -31,7 +34,17 @@ following data:
 
 ### Mobile
 
-You can run the integration tests using the `flutter test` command:
+You can run the Android integration tests using the [`patrol_cli`](https://pub.dev/packages/patrol_cli) command:
+
+```sh
+patrol test \
+  --flavor prod \
+  --dart-define USER_1_EMAIL="EMAIL" \
+  --dart-define USER_1_PASSWORD="PASSWORD" \
+  -t integration_test/app_test.dart
+```
+
+You can run the integration tests for the other platforms using the [`patrol_cli`](https://pub.dev/packages/patrol_cli) command:
 
 ```sh
 fvm flutter test \