Skip to content

Commit 174aa4c

Browse files
test: refactors the e2e sh script to mjs for flakiness detection (#19838)
<!-- Please submit this PR as a draft initially. Do not mark it as "Ready for review" until the template has been completely filled out, and PR status checks have passed at least once. --> ## **Description** - Replace bash e2e runner with Node ESM. - Add per-split duplication for new/modified specs to detect flakiness (base + 2 retries). - Remove Octokit dependency; use native fetch + GitHub GraphQL. **What changed** - run-e2e-tags-gha.mjs: runs tests based on the Test Suite Tag passed on + ensures new and modified tests run 3 times to avoid the introduction of flaky tests **Behavior** - If skip-e2e-quality-gate label present: no test duplication. - If no PR or no changed specs: no duplication. - duplication only for ADDED/MODIFIED specs selected in the current split. **Testing** A new dummy test was added in order to showcase the spec file duplication. Example logs from this [run](https://github.com/MetaMask/metamask-mobile/actions/runs/17773198289/job/50515597677?pr=19838) (no label applied and a new added test): ``` 🚀 Running wallet-platform-android-smoke-2 tests on android Using TEST_SUITE_TAG: SmokeWalletPlatform 🚀 Starting E2E tests... Running split 2 of 2 Searching for tests with pattern: SmokeWalletPlatform Running split 2 of 2 📋 Found 9 matching test files in total 🧪 Duplicated for flakiness check: e2e/specs/multisrp/dummy-flakyness-retry-1.spec.ts 🧪 Duplicated for flakiness check: e2e/specs/multisrp/dummy-flakyness-retry-2.spec.ts 🧪 After duplication (per split), total selected files: 6 🔍 Running 6 tests for split 2 of 2: - e2e/specs/multisrp/dummy-flakyness.spec.ts --> Original test file - e2e/specs/multisrp/dummy-flakyness-retry-1.spec.ts --> Retry 1 - e2e/specs/multisrp/dummy-flakyness-retry-2.spec.ts --> Retry 2 - e2e/specs/multisrp/export-srp-from-account-actions.spec.ts - e2e/specs/multisrp/export-srp-from-settings.spec.ts - e2e/specs/wallet/incoming-transactions.spec.ts 🚀 Running matching tests for split 2... 🤖 Running Android tests on GitHub Actions yarn run v1.22.22 $ IS_TEST='true' NODE_OPTIONS='--experimental-vm-modules' detox test -c android.github_ci.release --headless --record-logs all e2e/specs/multisrp/dummy-flakyness.spec.ts e2e/specs/multisrp/dummy-flakyness-retry-1.spec.ts e2e/specs/multisrp/dummy-flakyness-retry-2.spec.ts e2e/specs/multisrp/export-srp-from-account-actions.spec.ts e2e/specs/multisrp/export-srp-from-settings.spec.ts e2e/specs/wallet/incoming-transactions.spec.ts ``` Example logs from this [run](https://github.com/MetaMask/metamask-mobile/actions/runs/17774080179/job/50518441475?pr=19838) (with label applied): ``` 🚀 Running wallet-platform-android-smoke-2 tests on android Using TEST_SUITE_TAG: SmokeWalletPlatform 🚀 Starting E2E tests... Running split 2 of 2 Searching for tests with pattern: SmokeWalletPlatform Running split 2 of 2 📋 Found 9 matching test files in total ⏭️ skip-e2e-quality-gate detected; skipping flaky duplication 🔍 Running 4 tests for split 2 of 2: - e2e/specs/multisrp/dummy-flakyness.spec.ts - e2e/specs/multisrp/export-srp-from-account-actions.spec.ts - e2e/specs/multisrp/export-srp-from-settings.spec.ts - e2e/specs/wallet/incoming-transactions.spec.ts 🚀 Running matching tests for split 2... 🤖 Running Android tests on GitHub Actions yarn run v1.22.22 $ IS_TEST='true' NODE_OPTIONS='--experimental-vm-modules' detox test -c android.github_ci.release --headless --record-logs all e2e/specs/multisrp/dummy-flakyness.spec.ts e2e/specs/multisrp/export-srp-from-account-actions.spec.ts e2e/specs/multisrp/export-srp-from-settings.spec.ts e2e ``` <!-- Write a short description of the changes included in this pull request, also include relevant motivation and context. Have in mind the following questions: 1. What is the reason for the change? 2. What is the improvement/solution? --> ## **Changelog** <!-- If this PR is not End-User-Facing and should not show up in the CHANGELOG, you can choose to either: 1. Write `CHANGELOG entry: null` 2. Label with `no-changelog` If this PR is End-User-Facing, please write a short User-Facing description in the past tense like: `CHANGELOG entry: Added a new tab for users to see their NFTs` `CHANGELOG entry: Fixed a bug that was causing some NFTs to flicker` (This helps the Release Engineer do their job more quickly and accurately) --> CHANGELOG entry: ## **Related issues** Fixes: ## **Manual testing steps** ```gherkin Feature: my feature name Scenario: user [verb for user action] Given [describe expected initial app state] When user [verb for user action] Then [describe expected outcome] ``` ## **Screenshots/Recordings** <!-- If applicable, add screenshots and/or recordings to visualize the before and after of your change. --> ### **Before** <!-- [screenshots/recordings] --> ### **After** <!-- [screenshots/recordings] --> ## **Pre-merge author checklist** - [ ] I’ve followed [MetaMask Contributor Docs](https://github.com/MetaMask/contributor-docs) and [MetaMask Mobile Coding Standards](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/CODING_GUIDELINES.md). - [ ] I've completed the PR template to the best of my ability - [ ] I’ve included tests if applicable - [ ] I’ve documented my code using [JSDoc](https://jsdoc.app/) format if applicable - [ ] I’ve applied the right labels on the PR (see [labeling guidelines](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/LABELING_GUIDELINES.md)). Not required for external contributors. ## **Pre-merge reviewer checklist** - [ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed). - [ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.
1 parent 3143c90 commit 174aa4c

File tree

8 files changed

+483
-105
lines changed

8 files changed

+483
-105
lines changed

.github/scripts/run-e2e-tags-gha.mjs

Lines changed: 430 additions & 0 deletions
Large diffs are not rendered by default.

.github/workflows/ci.yml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -214,7 +214,7 @@ jobs:
214214
215215
needs_e2e_build:
216216
uses: ./.github/workflows/needs-e2e-build.yml
217-
217+
218218
build-android-apks:
219219
name: "Build Android APKs"
220220
if: ${{ github.event_name != 'merge_group' && needs.needs_e2e_build.outputs.android_changed == 'true' }}
@@ -233,6 +233,8 @@ jobs:
233233
id-token: write
234234
needs: [needs_e2e_build, build-android-apks]
235235
uses: ./.github/workflows/run-e2e-smoke-tests-android.yml
236+
with:
237+
changed_files: ${{ needs.needs_e2e_build.outputs.changed_files }}
236238
secrets: inherit
237239

238240
js-bundle-size-check:

.github/workflows/needs-e2e-build.yml

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,9 @@ on:
1919
builds:
2020
description: 'Whether any builds will happen'
2121
value: ${{ jobs.needs-e2e-build.outputs.builds }}
22+
changed_files:
23+
description: 'Changed files'
24+
value: ${{ jobs.needs-e2e-build.outputs.changed_files }}
2225

2326
jobs:
2427
needs-e2e-build:
@@ -28,6 +31,7 @@ jobs:
2831
android: ${{ steps.set-outputs.outputs.android_final }}
2932
ios: ${{ steps.set-outputs.outputs.ios_final }}
3033
builds: ${{ steps.set-outputs.outputs.builds }}
34+
changed_files: ${{ steps.set-outputs.outputs.changed_files }}
3135
env:
3236
# For a `pull_request` event, the head commit hash is `github.event.pull_request.head.sha`.
3337
# For a `push` event, the head commit hash is `github.sha`.
@@ -38,7 +42,7 @@ jobs:
3842
with:
3943
# Specifying `ref` ensures that the head commit is checked out directly.
4044
ref: ${{ env.HEAD_COMMIT_HASH }}
41-
45+
4246
- name: Check skip commit tag
4347
id: skip-tag
4448
run: |
@@ -114,6 +118,11 @@ jobs:
114118
- name: Set final outputs
115119
id: set-outputs
116120
run: |
121+
{
122+
echo "changed_files<<EOF"
123+
echo "${{ steps.filter.outputs.catch_all_files }}"
124+
echo "EOF"
125+
} >> "$GITHUB_OUTPUT"
117126
118127
# Skip if [skip-e2e] tag found or label exists on PR
119128
if [[ "${{ steps.skip-tag.outputs.SKIP }}" == "true" ]] || [[ "${{ steps.skip-label.outputs.SKIP }}" == "true" ]]; then
@@ -187,4 +196,4 @@ jobs:
187196
else
188197
builds="false"
189198
fi
190-
echo "builds=$builds" >> "${GITHUB_OUTPUT}"
199+
echo "builds=$builds" >> "${GITHUB_OUTPUT}"

.github/workflows/run-e2e-smoke-tests-android.yml

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,12 @@ name: Android E2E Smoke Tests
22

33
on:
44
workflow_call:
5+
inputs:
6+
changed_files:
7+
description: 'Changed files'
8+
required: false
9+
type: string
10+
default: ''
511

612
permissions:
713
contents: read
@@ -20,6 +26,7 @@ jobs:
2026
test_suite_tag: "SmokeConfirmations"
2127
split_number: ${{ matrix.split }}
2228
total_splits: 3
29+
changed_files: ${{ inputs.changed_files }}
2330
secrets: inherit
2431

2532
trade-android-smoke:
@@ -34,6 +41,7 @@ jobs:
3441
test_suite_tag: "SmokeTrade"
3542
split_number: ${{ matrix.split }}
3643
total_splits: 2
44+
changed_files: ${{ inputs.changed_files }}
3745
secrets: inherit
3846

3947
wallet-platform-android-smoke:
@@ -48,6 +56,7 @@ jobs:
4856
test_suite_tag: "SmokeWalletPlatform"
4957
split_number: ${{ matrix.split }}
5058
total_splits: 2
59+
changed_files: ${{ inputs.changed_files }}
5160
secrets: inherit
5261

5362
identity-android-smoke:
@@ -62,6 +71,7 @@ jobs:
6271
test_suite_tag: "SmokeIdentity"
6372
split_number: ${{ matrix.split }}
6473
total_splits: 2
74+
changed_files: ${{ inputs.changed_files }}
6575
secrets: inherit
6676

6777
accounts-android-smoke:
@@ -76,6 +86,7 @@ jobs:
7686
test_suite_tag: "SmokeAccounts"
7787
split_number: ${{ matrix.split }}
7888
total_splits: 2
89+
changed_files: ${{ inputs.changed_files }}
7990
secrets: inherit
8091

8192
network-abstraction-android-smoke:
@@ -90,6 +101,7 @@ jobs:
90101
test_suite_tag: "SmokeNetworkAbstractions"
91102
split_number: ${{ matrix.split }}
92103
total_splits: 2
104+
changed_files: ${{ inputs.changed_files }}
93105
secrets: inherit
94106

95107
network-expansion-android-smoke:
@@ -104,6 +116,7 @@ jobs:
104116
test_suite_tag: "SmokeNetworkExpansion"
105117
split_number: ${{ matrix.split }}
106118
total_splits: 2
119+
changed_files: ${{ inputs.changed_files }}
107120
secrets: inherit
108121

109122
confirmations-redesigned-android-smoke:
@@ -118,6 +131,7 @@ jobs:
118131
test_suite_tag: "SmokeConfirmationsRedesigned"
119132
split_number: ${{ matrix.split }}
120133
total_splits: 2
134+
changed_files: ${{ inputs.changed_files }}
121135
secrets: inherit
122136

123137
report-android-smoke-tests:

.github/workflows/run-e2e-smoke-tests-ios.yml

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,12 @@ name: iOS E2E Smoke Tests
22

33
on:
44
workflow_call:
5+
inputs:
6+
changed_files:
7+
description: 'Changed files'
8+
required: false
9+
type: string
10+
default: ''
511

612
permissions:
713
contents: read
@@ -20,6 +26,7 @@ jobs:
2026
test_suite_tag: "SmokeConfirmations"
2127
split_number: ${{ matrix.split }}
2228
total_splits: 3
29+
changed_files: ${{ inputs.changed_files }}
2330
secrets: inherit
2431

2532
confirmations-redesigned-ios-smoke:
@@ -34,6 +41,7 @@ jobs:
3441
test_suite_tag: "SmokeConfirmationsRedesigned"
3542
split_number: ${{ matrix.split }}
3643
total_splits: 2
44+
changed_files: ${{ inputs.changed_files }}
3745
secrets: inherit
3846

3947
trade-ios-smoke:
@@ -48,6 +56,7 @@ jobs:
4856
test_suite_tag: "SmokeTrade"
4957
split_number: ${{ matrix.split }}
5058
total_splits: 2
59+
changed_files: ${{ inputs.changed_files }}
5160
secrets: inherit
5261

5362
wallet-platform-ios-smoke:
@@ -62,6 +71,7 @@ jobs:
6271
test_suite_tag: "SmokeWalletPlatform"
6372
split_number: ${{ matrix.split }}
6473
total_splits: 2
74+
changed_files: ${{ inputs.changed_files }}
6575
secrets: inherit
6676

6777
identity-ios-smoke:
@@ -76,6 +86,7 @@ jobs:
7686
test_suite_tag: "SmokeIdentity"
7787
split_number: ${{ matrix.split }}
7888
total_splits: 2
89+
changed_files: ${{ inputs.changed_files }}
7990
secrets: inherit
8091

8192
accounts-ios-smoke:
@@ -90,6 +101,7 @@ jobs:
90101
test_suite_tag: "SmokeAccounts"
91102
split_number: ${{ matrix.split }}
92103
total_splits: 2
104+
changed_files: ${{ inputs.changed_files }}
93105
secrets: inherit
94106

95107
network-abstraction-ios-smoke:
@@ -104,6 +116,7 @@ jobs:
104116
test_suite_tag: "SmokeNetworkAbstractions"
105117
split_number: ${{ matrix.split }}
106118
total_splits: 2
119+
changed_files: ${{ inputs.changed_files }}
107120
secrets: inherit
108121

109122
network-expansion-ios-smoke:
@@ -118,6 +131,7 @@ jobs:
118131
test_suite_tag: "SmokeNetworkExpansion"
119132
split_number: ${{ matrix.split }}
120133
total_splits: 2
134+
changed_files: ${{ inputs.changed_files }}
121135
secrets: inherit
122136

123137
report-ios-smoke-tests:

.github/workflows/run-e2e-workflow.yml

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,11 @@ on:
3333
required: false
3434
type: number
3535
default: 30
36+
changed_files:
37+
description: 'Changed files'
38+
required: false
39+
type: string
40+
default: ''
3641

3742
jobs:
3843
test-e2e-mobile:
@@ -46,7 +51,10 @@ jobs:
4651
PREBUILT_IOS_APP_PATH: artifacts/MetaMask.app
4752
METAMASK_ENVIRONMENT: 'qa'
4853
METAMASK_BUILD_TYPE: 'main'
54+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
55+
PLATFORM: ${{ inputs.platform }}
4956
TEST_SUITE_TAG: ${{ inputs.test_suite_tag }}
57+
CHANGED_FILES: ${{ inputs.changed_files }}
5058
GITHUB_CI: 'true'
5159
SPLIT_NUMBER: ${{ inputs.split_number }}
5260
TOTAL_SPLITS: ${{ inputs.total_splits }}
@@ -172,16 +180,11 @@ jobs:
172180
173181
# Run tests (Detox/Jest handle retries internally)
174182
echo "🚀 Starting E2E tests..."
175-
if [[ "$platform" == "ios" ]]; then
176-
export BITRISE_TRIGGERED_WORKFLOW_ID="ios_workflow"
177-
else
178-
export BITRISE_TRIGGERED_WORKFLOW_ID="android_workflow"
179-
fi
180183
181184
# Always use the splitting script (handles both split and non-split cases)
182185
echo "Running split ${{ inputs.split_number }} of ${{ inputs.total_splits }}"
183186
184-
./scripts/run-e2e-tags-gha.sh
187+
node .github/scripts/run-e2e-tags-gha.mjs
185188
186189
echo "✅ Test execution completed"
187190
env:

.github/workflows/temp-ios-workflow.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,4 +38,6 @@ jobs:
3838
id-token: write
3939
needs: [needs_e2e_build, build-ios-apps]
4040
uses: ./.github/workflows/run-e2e-smoke-tests-ios.yml
41+
with:
42+
changed_files: ${{ needs.needs_e2e_build.outputs.changed_files }}
4143
secrets: inherit

scripts/run-e2e-tags-gha.sh

Lines changed: 0 additions & 96 deletions
This file was deleted.

0 commit comments

Comments
 (0)