[Mobile] - Try splitting E2E tests #47814
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
geriux
added
[Status] In Progress
|
Size Change: -83 B (0%) Total Size: 1.31 MB
ℹ️ View Unchanged
|
|
Flaky tests detected in 3ff8caf. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/4114842566
|
…on CI. It also changes the approach on iOS to also scroll to get the last block if needed. After the React Native upgrade the block list doesn't render the whole thing at the same time so on iOS it was causing flakiness due to trying to get the last block without scrolling or waiting for all content to be there.
geriux
force-pushed
the
rnmobile/split-e2e-tests
branch
from
d00944e
to
e00cba1
Compare
geriux
added
the
Mobile App - Automation
fluiddot
reviewed
Feb 7, 2023
packages/react-native-editor/__device-tests__/gutenberg-editor-rendering-media-blocks.test.js
Outdated
Show resolved
Hide resolved
packages/react-native-editor/__device-tests__/pages/editor-page.js
Outdated
Show resolved
Hide resolved
As you mentioned @geriux , looks like the flakiness could be mostly caused by the fact that the initial HTML E2E tests are being run on slow machines. However, this might be the signal that performance regressions might have been introduced recently. Since performance issues and tooling to identify this type of regressions will be introduced in the future, as they are being worked on in a different project, I'd stick with this approach especially to make the mobile E2E tests more stable. |
fluiddot
approved these changes
Feb 7, 2023
geriux
removed
the
Mobile App - Automation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related PRs:
What?
This PR splits the only E2E test we have in this repo into three.
Why?
In the past, we've had issues when incrementing the content of the
initial-HTMLfile for testing. I can't remember the goal for this test but I think it's mostly to check all blocks render correctly, it could also be to stress test the editor by adding a lot of content but I don't think this is the case.After the React Native upgrade I believe we've started to have more flakiness on the iOS tests, I think this is because the rendering of the block list is a bit different, where it starts rendering content as batches and before it blocked the UI until It finished rendering the blocks.
For the
gutenberg-editor-initial-html.test.json iOS, we were expecting to get the last block right after setting the content. On Android, we scroll down up to the very end to find the block.How?
First, we split the
gutenberg-editor-initial-html.test.jsinto three for this repo consisting on:All of those are in separate tests, for all of them, it would use the Android approach to scroll down to the end to get the
Add Paragraph blockplaceholder. It's worth noting that for iOS sometimes scrolling down won't be needed as it would find the block right away.There's a new helper
scrollAndReturnElementByAccessibilityIdwhich is similar toandroidScrollAndReturnElementbut it's for both platforms and it looks for an accessibility ID instead.For this repo we will only run those tests, we are updating the filter to take into account tests that have
renderingas part of the names.The initial HTML test will only run on Gutenberg Mobile, the reasoning for this is that there we have more powerful machines, whereas here I think we are using the most basic ones limiting the performance of the simulators.
Testing Instructions
CI Checks should pass on both Gutenberg and Gutenberg mobile. Try re-running the E2E tests to see if they succeed again.
Testing Instructions for Keyboard
N/A
Screenshots or screencast
N/A