[1364] Run UI/integration tests on phone/tablet emulators

[mcomella]

Why/User Benefit/User Problem

Right now, we can't run our tests on CI. They get run at least once a day off of master. However, this means that the feedback cycle is delayed, it's unclear whose change broke the tests, and no one feels ownership over fixing them: they don't get fix. When the tests break, it is a sign that the user experience is broken (assuming the tests are reliable, and they should be!) so the bustage needs to be investigated!

We can run our tests on CI on phone/tablet emulators. However, because we intend to run on TV emulators, there are issues to solve.

What / Requirements

• Figure out how to open the overlay on phone/tablet emulators (menu button on these devices opens recent apps list)
• Identify and fix any other issues with UI/integration tests on these emulators
• Talk to QA about getting these tests run on CI

Acceptance Criteria (how do I know when I’m done?)

[mcomella]

Aaron has been looking into this.

[mcomella]

@athomasmoz I would like to work on this this sprint.

[athomasmoz]

Added to sprint. Can you please add an estimate and let me know how that impacts our current sprint commitment? Does it mean anything else will fall out?

[mcomella]

@athomasmoz Little impact on the current Echo Show sprint: I plan to prioritize the current Echo Show sprint and think I can finish that by the end of the week, barring any large distractions. As for Firefox for Fire TV, I don't know that I'll finish this by the end of the sprint but in that case, we can just bump it to the next one: I don't think it's something we need to commit to a release.

[AaronMT]

(@mcomella) there was one or two attempts by (@npark-mozilla) a couple months ago at loading a debug build on a Nexus tablet in the office and at the time the major blocker for tests passing was the interactions with the overlay/menu system and UI Automator/UiDevice interactions expecting remote usage (e.,g pressMenu()). We haven't looked at it since but I still expect there to be issues.

[nojunpark]

@mcomella So I ran basicnavigationtest on Moto G5 today, and ran into strictmode policy violation, but if I manually tap OK button to dismiss (had to do it 4 times), the test runs and passes. If we can fix this issue, we should be able to try the rest of the test on Firebase.

[nojunpark]

One more thing that we remembered is that on a low-resolution phone, the entire area of the app won't be drawn properly. (especially vertically) Initially we thought this may lead to the UI interaction failure, but it looks like the tests are passing nevertheless.

[liuche]

@mcomella we discussed this during sprint planning, and it looks like there are some more comments for this - so it looks like there are some more action items from QA.

[mcomella]

Background context: Severin disabled the tests in #1648 so we can get the tests running in CI: #615.

@baron-severin Do you have follow-ups filed to re-enable the tests or should we use this bug to re-enable the tests?

Also, can you look into the BasicNavigationTest issue (or should we just disable it)? It seems to be blocking #615.

---

@npark-mozilla Can you clarify why we are running tests on phone emulators and not tablet or TV emulators again? Also, is it possible to disable testing on low-resolution phones? I think we should just test on the largest phones available, assuming we have to test on phones.

[nojunpark]

@mcomella on Firebase, there is no TV emulator available. There is a Nexus 9 emulator (they deprecated the device), but from the Focus experience it is quite buggy.

We can choose to run tests on specific devices, but when I ran it on Pixel 2, which is supposedly a high res device, it still had screen drawn incorrectly.

[severinrudie]

@mcomella #1668, #1649, and #1360 have been filed as follow-ups for ignored tests.

BasicNavigationTest has already been ignored (it's tracked in #1649). I've also been working on fixing it.

[mcomella]

@npark-mozilla That's a shame: do we have to tie the testing to specific devices? e.g. can we use custom emulator screen sizes or the WXGA 10.1 emulator we used for developing FFES?

[nojunpark]

@mcomella so gcloud firebase test android models list shows the currently available firebase devices for testing, (and no... it doesn't let you create a custom emulator to run tests, if we're gonna do that, we need to do this all locally, which I don't recommend) But I just realized pixel 2 is only 1920 x1080. I see Nexus 6P is 2560 x1440. This is almost on par with Nexus 9, (2048 X1536) perhaps we should try this. I can trial run it on Firebase as soon as the strictmode error issue is resolved.

[mcomella]

@npark-mozilla Where did you get the gcloud executable from? Alternatively, feel free to send me a list of devices I can help look over too.

Note that resolution isn't the most important thing (because display density can collapse how large pixels are physical): screen size is. The 6P is more like the Pixel 2 XL in screen size.

I can trial run it on Firebase as soon as the strictmode error issue is resolved.

@baron-severin Have you seen that issue?

[nojunpark]

@mcomella sorry, forgot to attach this list: devicelist.txt Note that some devices only support older android versions

[severinrudie]

I haven't, no. @npark-mozilla do you have reproduction steps that we could follow to try and fix the strictmode error?

[nojunpark]

@baron-severin Oh, just run any Fire TV UI tests on android studio, on a phone device.
Interestingly, this doesn't fail on an emulator... Hmm. I am not sure whether we should ignore it or try to fix it still.

[severinrudie]

@npark-mozilla I'm trying on a Galaxy S7 on Android 7.0. Haven't encountered any strict mode violations, but I am seeing the following pop up occasionally (and it's causing tests to fail). Is this what you were seeing as well?

[nojunpark]

@baron-severin No I dont get this message - I suspect it's because of the phone's USB setting? (It's probably set to disable data transfer) Does your computer's adb detect the connected phone?

The strictmode failure I see happens when the test is just about to start.

[severinrudie]

@npark-mozilla the transfer mode was flipping itself back to data transfer, but it's an old, beat up device and may be a quirk specific to it. I wasn't able to reproduce what you reported, is there any chance you could send me a video? Otherwise I'm mostly blocked on this one.

[mcomella]

@mcomella sorry, forgot to attach this list: devicelist.txt Note that some devices only support older android versions

@npark-mozilla Nexus 9 is the flaky device? Have we considered the Nexus 10 or the Nexus 7? There's also the Galaxy Tab 3, but it's old, and various Samsung plus size devices (S9+, Note ..., etc.).

wrt Android version, ideally we run on 22 and 25 but I think any of Android 21+ will be okay. We may have to deal with issues on high API levels though (e.g. added permissions, changed ANR policies, etc.).

[nojunpark]

@mcomella since they only have API 22 and below, and I was kinda looking for API 25 (the way Espresso picks up DOM is kinda different between 25 and those below 25 I noticed), so haven't tried on those yet. But if 22 is okay, then we can give a try for sure. There's that nexus 7 clone that supports API 25 so maybe it's a good idea. @baron-severin I sent you the credentials for the remote testkit.

[mcomella]

(the way Espresso picks up DOM is kinda different between 25 and those

below 25 I noticed) I haven't personally noticed this but it's possible – the WebView version is frozen with the API version so older emulators are using older WebView versions. In any case, it's worth a shot! :) The WebView version for Fire TV is frozen at v59 currently, which is maybe 1.5 years old, iirc. I'd guess those emulators would be a little older.

[severinrudie]

Update re the strictmode violation. This is caused by an android.os.strictmode.UntaggedSocketViolation thrown by OkHttp3. They have marked the issue as will not fix, and suggest either disabling this check, or adding some workaround code pasted into the ticket. A few people have responded saying that the workaround code does not work.

I'm not yet sure why this fails on certain Remote TestKit devices, but not on our own emulators or devices. Still investigating as to the best solution.

See: square/okhttp#3537

EDIT: specifically, this check falls under StrictMode.ThreadPolicy#detectNetwork. From the linked discussion it sounds like the failing check is pretty useless, but other checks that we do want may well fall into the same category.

[mcomella]

fwiw, we could consider logging these StrictMode events instead of displaying a user-facing dialog. User-facing is usually good because they're more noticeable, but not in cases like these. x_x