Sanity tests robot pattern and reliability refactor #4420

ouchadam · 2021-11-05T15:01:25Z

Ports the sanity test to the robot pattern (extracting the UI navigation and manipulation to dedicated classes which return themselves back to the original state), this enables fine grain reusable navigation
Swaps out timeouts for view waits where possible
Adds missing view waits to make the tests less flaky (they now consistently pass for me with an emulator)
Disables the sign out flow as the tests currently hit a bug

GIF	SIGN OUT BUG

- will allow reuse of navigating around the app (eg enabling develop options and go back to previous locations)

- improves the flakyness around the bottom sheets and dialogs

…moved the flakyness when running the tests on my emulator

…t robot

- also makes waitForActivity stronger by also waiting for child views

…it for view

ouchadam · 2021-11-05T15:09:52Z

vector/src/androidTest/java/im/vector/app/EspressoExt.kt

@@ -70,6 +73,8 @@ fun waitForView(viewMatcher: Matcher<View>, timeout: Long = 10_000, waitForDispl
            val endTime = startTime + timeout
            val visibleMatcher = isDisplayed()

+            uiController.loopMainThreadForAtLeast(100)


I really wanted to avoid this but this single change made the biggest difference for me when it came to reliability.

Dialogs and bottomsheets would be visible but not interactable, back actions or button presses would be ignored half the time~.

bmarty

Thanks for this nice rework.
A few feedbacks

bmarty · 2021-11-05T15:06:54Z

vector/src/androidTest/java/im/vector/app/ui/UiAllScreensSanityTest.kt

+// Disable until the "you don't have a session for id %d" sign out bug is fixed
+//        elementRobot.signout()
+////        Login again on the same account
+//        elementRobot.login(userId)


We do not test regular login then... This is a bit annoying

agreed! I'd like to look into the bug and reenable this ASAP

bmarty · 2021-11-05T15:08:23Z

vector/src/androidTest/java/im/vector/app/ui/robot/CreateNewRoomRobot.kt

+import im.vector.app.features.home.room.detail.RoomDetailActivity
+
+class CreateNewRoomRobot(
+        var createdRoom: Boolean = false


What is the goal of this boolean? (I do not know the Robot pattern)

it's to avoid a pressBack if we create a new room as navigating to the new room finishes the current room creation activity (I hope that makes sense? 😅 )

OK, I see. But this test should always create a room, so this is still strange to me.

bmarty · 2021-11-05T15:10:45Z

vector/src/androidTest/java/im/vector/app/ui/robot/NewRoomRobot.kt

+        BaristaClickInteractions.clickOn(R.string.create_new_room)
+        val createNewRoomRobot = CreateNewRoomRobot()
+        block(createNewRoomRobot)
+        if (!createNewRoomRobot.createdRoom) {


the test should fail in this case, no?

without this the next steps will fail as every robot assumes the start and end position are the same

in the case of creating a new room the room creation screen is automatically finished so there's no need to press back, if we did then we'd exit the app as the RoomDetails

// move to room list tab and press #+ elementRobot.newRoom { // press create new room launches CreateRoomActivity createNewRoom { // press create, navigates to the RoomDetailsActivity, CreateRoomActivity automatically closes createRoom { // goes through all the room detail checks crawl() // on createRoom finish pressBack to take us back to root activity } // avoid pressing back because we created a room } // avoid pressing back because we created a room }

ouchadam · 2021-11-05T15:15:08Z

some observations during my testing

Sometimes I see a concurrent exception during the login flow, seems to be coming from the testing library

2021-11-05 14:45:28.209 12441-12441/im.vector.app.debug E/MonitoringInstr: Exception encountered by: im.vector.app.features.login.LoginActivity@c5555e9. Dumping thread state to outputs and pining for the fjords.
    java.util.ConcurrentModificationException
        at java.util.ArrayList$Itr.next(ArrayList.java:860)
        at androidx.test.internal.runner.lifecycle.ActivityLifecycleMonitorImpl.signalLifecycleChange(ActivityLifecycleMonitorImpl.java:159)
        at androidx.test.runner.MonitoringInstrumentation.callActivityOnStop(MonitoringInstrumentation.java:754)
        at android.app.Activity.performStop(Activity.java:8018)
        at android.app.ActivityThread.callActivityOnStop(ActivityThread.java:4616)
        at android.app.ActivityThread.performStopActivityInner(ActivityThread.java:4594)
        at android.app.ActivityThread.handleStopActivity(ActivityThread.java:4669)
        at android.app.servertransaction.TransactionExecutor.performLifecycleSequence(TransactionExecutor.java:233)
        at android.app.servertransaction.TransactionExecutor.cycleToPath(TransactionExecutor.java:201)
        at android.app.servertransaction.TransactionExecutor.executeLifecycleState(TransactionExecutor.java:173)
        at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:97)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2016)
        at android.os.Handler.dispatchMessage(Handler.java:107)
        at android.os.Looper.loop(Looper.java:214)

On my emulator the EditTextPreference causes an OS tombstone, this happens in the display name dialog but also when viewing the System android build version 🤔

github-actions · 2021-11-05T15:30:21Z

Unit Test Results

  62 files ±0   62 suites ±0 1m 9s ⏱️ +4s
118 tests ±0 118 ✔️ ±0 0 💤 ±0 0 ❌ ±0
350 runs ±0 350 ✔️ ±0 0 💤 ±0 0 ❌ ±0

Results for commit 78675d4. ± Comparison against base commit be932a8.

♻️ This comment has been updated with latest results.

bmarty

Thanks for the update.
I'll trust your judgment for the boolean var in the room creation robot.

bmarty · 2021-11-08T14:08:03Z

ktlint issues fixed on develop in 2b58c0e

ouchadam added 16 commits November 4, 2021 12:35

adding login timeouts to reduce flakyness during login

ed4f26d

using robot pattern to navigate the settings

2f3716d

- will allow reuse of navigating around the app (eg enabling develop options and go back to previous locations)

porting settings crawling to the robot pattern

67fe07a

porting the new direct message checks to the robot pattern

e800c59

porting the room flows to the robot pattern

557609c

adding helper for waiting until view is ready

5284765

removing sleeps in favour of waiting for the views to become ready

e0544a6

- improves the flakyness around the bottom sheets and dialogs

replacing sleeps with view waits where possible

f08d396

making all waitForViews have an initial 100ms wait this completely re…

d2aec09

…moved the flakyness when running the tests on my emulator

ensuring we end up back where we started via the root robot helpers

17789f0

porting ui base to be a onboarding robot and providing via the elemen…

0f1c9f4

…t robot

moving the signout to the robot but commenting out whilst a bug is fixed

32f5893

- also makes waitForActivity stronger by also waiting for child views

moving settings robots to their own pacakge

264f49b

removing no longer needed sleeps

7b6773f

using static imports for the espresso helpers and reusing existing wa…

803929e

…it for view

adding missing activity wait whilst navigating to the room members

0405c2d

ouchadam commented Nov 5, 2021

View reviewed changes

bmarty reviewed Nov 5, 2021

View reviewed changes

ouchadam added 3 commits November 5, 2021 15:35

relying on createRoom to close itself rather than the newRoom flow

b1397ae

ensuring we wait for the create room fragment to be ready

657a613

fixing formatting

78675d4

bmarty approved these changes Nov 8, 2021

View reviewed changes

bmarty merged commit cea2206 into develop Nov 8, 2021

bmarty deleted the feature/adm/sanity-check-robot branch November 8, 2021 13:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sanity tests robot pattern and reliability refactor #4420

Sanity tests robot pattern and reliability refactor #4420

ouchadam commented Nov 5, 2021 •

edited

Loading

ouchadam Nov 5, 2021

bmarty left a comment

bmarty Nov 5, 2021

ouchadam Nov 5, 2021

bmarty Nov 5, 2021

ouchadam Nov 5, 2021 •

edited

Loading

bmarty Nov 8, 2021

bmarty Nov 5, 2021

ouchadam Nov 5, 2021

ouchadam commented Nov 5, 2021

github-actions bot commented Nov 5, 2021 •

edited

Loading

bmarty left a comment

bmarty commented Nov 8, 2021

Sanity tests robot pattern and reliability refactor #4420

Sanity tests robot pattern and reliability refactor #4420

Conversation

ouchadam commented Nov 5, 2021 • edited Loading

ouchadam Nov 5, 2021

Choose a reason for hiding this comment

bmarty left a comment

Choose a reason for hiding this comment

bmarty Nov 5, 2021

Choose a reason for hiding this comment

ouchadam Nov 5, 2021

Choose a reason for hiding this comment

bmarty Nov 5, 2021

Choose a reason for hiding this comment

ouchadam Nov 5, 2021 • edited Loading

Choose a reason for hiding this comment

bmarty Nov 8, 2021

Choose a reason for hiding this comment

bmarty Nov 5, 2021

Choose a reason for hiding this comment

ouchadam Nov 5, 2021

Choose a reason for hiding this comment

ouchadam commented Nov 5, 2021

github-actions bot commented Nov 5, 2021 • edited Loading

Unit Test Results

bmarty left a comment

Choose a reason for hiding this comment

bmarty commented Nov 8, 2021

ouchadam commented Nov 5, 2021 •

edited

Loading

ouchadam Nov 5, 2021 •

edited

Loading

github-actions bot commented Nov 5, 2021 •

edited

Loading