Commit 174aa4c
authored
test: refactors the e2e sh script to mjs for flakiness detection (#19838)
<!--
Please submit this PR as a draft initially.
Do not mark it as "Ready for review" until the template has been
completely filled out, and PR status checks have passed at least once.
-->
## **Description**
- Replace bash e2e runner with Node ESM.
- Add per-split duplication for new/modified specs to detect flakiness
(base + 2 retries).
- Remove Octokit dependency; use native fetch + GitHub GraphQL.
**What changed**
- run-e2e-tags-gha.mjs: runs tests based on the Test Suite Tag passed on
+ ensures new and modified tests run 3 times to avoid the introduction
of flaky tests
**Behavior**
- If skip-e2e-quality-gate label present: no test duplication.
- If no PR or no changed specs: no duplication.
- duplication only for ADDED/MODIFIED specs selected in the current
split.
**Testing**
A new dummy test was added in order to showcase the spec file
duplication.
Example logs from this
[run](https://github.com/MetaMask/metamask-mobile/actions/runs/17773198289/job/50515597677?pr=19838)
(no label applied and a new added test):
```
🚀 Running wallet-platform-android-smoke-2 tests on android
Using TEST_SUITE_TAG: SmokeWalletPlatform
🚀 Starting E2E tests...
Running split 2 of 2
Searching for tests with pattern: SmokeWalletPlatform
Running split 2 of 2
📋 Found 9 matching test files in total
🧪 Duplicated for flakiness check: e2e/specs/multisrp/dummy-flakyness-retry-1.spec.ts
🧪 Duplicated for flakiness check: e2e/specs/multisrp/dummy-flakyness-retry-2.spec.ts
🧪 After duplication (per split), total selected files: 6
🔍 Running 6 tests for split 2 of 2:
- e2e/specs/multisrp/dummy-flakyness.spec.ts --> Original test file
- e2e/specs/multisrp/dummy-flakyness-retry-1.spec.ts --> Retry 1
- e2e/specs/multisrp/dummy-flakyness-retry-2.spec.ts --> Retry 2
- e2e/specs/multisrp/export-srp-from-account-actions.spec.ts
- e2e/specs/multisrp/export-srp-from-settings.spec.ts
- e2e/specs/wallet/incoming-transactions.spec.ts
🚀 Running matching tests for split 2...
🤖 Running Android tests on GitHub Actions
yarn run v1.22.22
$ IS_TEST='true' NODE_OPTIONS='--experimental-vm-modules' detox test -c android.github_ci.release --headless --record-logs all e2e/specs/multisrp/dummy-flakyness.spec.ts e2e/specs/multisrp/dummy-flakyness-retry-1.spec.ts e2e/specs/multisrp/dummy-flakyness-retry-2.spec.ts e2e/specs/multisrp/export-srp-from-account-actions.spec.ts e2e/specs/multisrp/export-srp-from-settings.spec.ts e2e/specs/wallet/incoming-transactions.spec.ts
```
Example logs from this
[run](https://github.com/MetaMask/metamask-mobile/actions/runs/17774080179/job/50518441475?pr=19838)
(with label applied):
```
🚀 Running wallet-platform-android-smoke-2 tests on android
Using TEST_SUITE_TAG: SmokeWalletPlatform
🚀 Starting E2E tests...
Running split 2 of 2
Searching for tests with pattern: SmokeWalletPlatform
Running split 2 of 2
📋 Found 9 matching test files in total
⏭️ skip-e2e-quality-gate detected; skipping flaky duplication
🔍 Running 4 tests for split 2 of 2:
- e2e/specs/multisrp/dummy-flakyness.spec.ts
- e2e/specs/multisrp/export-srp-from-account-actions.spec.ts
- e2e/specs/multisrp/export-srp-from-settings.spec.ts
- e2e/specs/wallet/incoming-transactions.spec.ts
🚀 Running matching tests for split 2...
🤖 Running Android tests on GitHub Actions
yarn run v1.22.22
$ IS_TEST='true' NODE_OPTIONS='--experimental-vm-modules' detox test -c android.github_ci.release --headless --record-logs all e2e/specs/multisrp/dummy-flakyness.spec.ts e2e/specs/multisrp/export-srp-from-account-actions.spec.ts e2e/specs/multisrp/export-srp-from-settings.spec.ts e2e
```
<!--
Write a short description of the changes included in this pull request,
also include relevant motivation and context. Have in mind the following
questions:
1. What is the reason for the change?
2. What is the improvement/solution?
-->
## **Changelog**
<!--
If this PR is not End-User-Facing and should not show up in the
CHANGELOG, you can choose to either:
1. Write `CHANGELOG entry: null`
2. Label with `no-changelog`
If this PR is End-User-Facing, please write a short User-Facing
description in the past tense like:
`CHANGELOG entry: Added a new tab for users to see their NFTs`
`CHANGELOG entry: Fixed a bug that was causing some NFTs to flicker`
(This helps the Release Engineer do their job more quickly and
accurately)
-->
CHANGELOG entry:
## **Related issues**
Fixes:
## **Manual testing steps**
```gherkin
Feature: my feature name
Scenario: user [verb for user action]
Given [describe expected initial app state]
When user [verb for user action]
Then [describe expected outcome]
```
## **Screenshots/Recordings**
<!-- If applicable, add screenshots and/or recordings to visualize the
before and after of your change. -->
### **Before**
<!-- [screenshots/recordings] -->
### **After**
<!-- [screenshots/recordings] -->
## **Pre-merge author checklist**
- [ ] I’ve followed [MetaMask Contributor
Docs](https://github.com/MetaMask/contributor-docs) and [MetaMask Mobile
Coding
Standards](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/CODING_GUIDELINES.md).
- [ ] I've completed the PR template to the best of my ability
- [ ] I’ve included tests if applicable
- [ ] I’ve documented my code using [JSDoc](https://jsdoc.app/) format
if applicable
- [ ] I’ve applied the right labels on the PR (see [labeling
guidelines](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/LABELING_GUIDELINES.md)).
Not required for external contributors.
## **Pre-merge reviewer checklist**
- [ ] I've manually tested the PR (e.g. pull and build branch, run the
app, test code being changed).
- [ ] I confirm that this PR addresses all acceptance criteria described
in the ticket it closes and includes the necessary testing evidence such
as recordings and or screenshots.1 parent 3143c90 commit 174aa4c
File tree
8 files changed
+483
-105
lines changed- .github
- scripts
- workflows
- scripts
8 files changed
+483
-105
lines changedLarge diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
214 | 214 | | |
215 | 215 | | |
216 | 216 | | |
217 | | - | |
| 217 | + | |
218 | 218 | | |
219 | 219 | | |
220 | 220 | | |
| |||
233 | 233 | | |
234 | 234 | | |
235 | 235 | | |
| 236 | + | |
| 237 | + | |
236 | 238 | | |
237 | 239 | | |
238 | 240 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
22 | 25 | | |
23 | 26 | | |
24 | 27 | | |
| |||
28 | 31 | | |
29 | 32 | | |
30 | 33 | | |
| 34 | + | |
31 | 35 | | |
32 | 36 | | |
33 | 37 | | |
| |||
38 | 42 | | |
39 | 43 | | |
40 | 44 | | |
41 | | - | |
| 45 | + | |
42 | 46 | | |
43 | 47 | | |
44 | 48 | | |
| |||
114 | 118 | | |
115 | 119 | | |
116 | 120 | | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
117 | 126 | | |
118 | 127 | | |
119 | 128 | | |
| |||
187 | 196 | | |
188 | 197 | | |
189 | 198 | | |
190 | | - | |
| 199 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
5 | 11 | | |
6 | 12 | | |
7 | 13 | | |
| |||
20 | 26 | | |
21 | 27 | | |
22 | 28 | | |
| 29 | + | |
23 | 30 | | |
24 | 31 | | |
25 | 32 | | |
| |||
34 | 41 | | |
35 | 42 | | |
36 | 43 | | |
| 44 | + | |
37 | 45 | | |
38 | 46 | | |
39 | 47 | | |
| |||
48 | 56 | | |
49 | 57 | | |
50 | 58 | | |
| 59 | + | |
51 | 60 | | |
52 | 61 | | |
53 | 62 | | |
| |||
62 | 71 | | |
63 | 72 | | |
64 | 73 | | |
| 74 | + | |
65 | 75 | | |
66 | 76 | | |
67 | 77 | | |
| |||
76 | 86 | | |
77 | 87 | | |
78 | 88 | | |
| 89 | + | |
79 | 90 | | |
80 | 91 | | |
81 | 92 | | |
| |||
90 | 101 | | |
91 | 102 | | |
92 | 103 | | |
| 104 | + | |
93 | 105 | | |
94 | 106 | | |
95 | 107 | | |
| |||
104 | 116 | | |
105 | 117 | | |
106 | 118 | | |
| 119 | + | |
107 | 120 | | |
108 | 121 | | |
109 | 122 | | |
| |||
118 | 131 | | |
119 | 132 | | |
120 | 133 | | |
| 134 | + | |
121 | 135 | | |
122 | 136 | | |
123 | 137 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
5 | 11 | | |
6 | 12 | | |
7 | 13 | | |
| |||
20 | 26 | | |
21 | 27 | | |
22 | 28 | | |
| 29 | + | |
23 | 30 | | |
24 | 31 | | |
25 | 32 | | |
| |||
34 | 41 | | |
35 | 42 | | |
36 | 43 | | |
| 44 | + | |
37 | 45 | | |
38 | 46 | | |
39 | 47 | | |
| |||
48 | 56 | | |
49 | 57 | | |
50 | 58 | | |
| 59 | + | |
51 | 60 | | |
52 | 61 | | |
53 | 62 | | |
| |||
62 | 71 | | |
63 | 72 | | |
64 | 73 | | |
| 74 | + | |
65 | 75 | | |
66 | 76 | | |
67 | 77 | | |
| |||
76 | 86 | | |
77 | 87 | | |
78 | 88 | | |
| 89 | + | |
79 | 90 | | |
80 | 91 | | |
81 | 92 | | |
| |||
90 | 101 | | |
91 | 102 | | |
92 | 103 | | |
| 104 | + | |
93 | 105 | | |
94 | 106 | | |
95 | 107 | | |
| |||
104 | 116 | | |
105 | 117 | | |
106 | 118 | | |
| 119 | + | |
107 | 120 | | |
108 | 121 | | |
109 | 122 | | |
| |||
118 | 131 | | |
119 | 132 | | |
120 | 133 | | |
| 134 | + | |
121 | 135 | | |
122 | 136 | | |
123 | 137 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
36 | 41 | | |
37 | 42 | | |
38 | 43 | | |
| |||
46 | 51 | | |
47 | 52 | | |
48 | 53 | | |
| 54 | + | |
| 55 | + | |
49 | 56 | | |
| 57 | + | |
50 | 58 | | |
51 | 59 | | |
52 | 60 | | |
| |||
172 | 180 | | |
173 | 181 | | |
174 | 182 | | |
175 | | - | |
176 | | - | |
177 | | - | |
178 | | - | |
179 | | - | |
180 | 183 | | |
181 | 184 | | |
182 | 185 | | |
183 | 186 | | |
184 | | - | |
| 187 | + | |
185 | 188 | | |
186 | 189 | | |
187 | 190 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| 41 | + | |
| 42 | + | |
41 | 43 | | |
This file was deleted.
0 commit comments