feat: improved way to trigger mmi e2e tests #27932

albertolive · 2024-10-17T13:48:00Z

Description

Summary

This pull request introduces a conditional check in the CircleCI pipeline to determine whether MMI Playwright tests should be executed based on the presence of the team-mmi label or if a reviewer from the MetaMask/mmi team has been assigned to the pull request. This optimizes the CI workflow by running MMI-specific tests only when necessary.

Previously, the end-to-end tests were only triggered when the team-mmi label was added to a pull request. However, in practice, no one outside of our team was adding this label, which resulted in the end-to-end MMI tests never being executed.

We already have an automation in place to automatically add a reviewer from the MetaMask/mmi team when relevant pull requests are opened. This pull request leverages that automation to ensure that the MMI tests are automatically triggered when a reviewer is assigned, making the process more efficient and reducing manual steps.

Changes Implemented

MMI Trigger Check

Added a check-mmi-trigger step that runs a shell script to check if the pull request has either:
- A team-mmi label.
- A reviewer from the MetaMask/mmi team assigned.
If either condition is met, the MMI-specific tests will be triggered. Otherwise, the workflow halts and skips these tests.

Modified Workflow

The prep-build-test-mmi-playwright and test-e2e-mmi-playwright steps are now conditionally executed based on the result of the check-mmi-trigger step.
If the MMI trigger is not present, the workflow halts after the check-mmi-trigger step, improving pipeline efficiency by avoiding unnecessary test execution.

Shell Script (check_mmi_trigger.sh)

Added a script to check for the presence of the team-mmi label or the MetaMask/mmi reviewer in a pull request.
The script sets the run_mmi_tests environment variable to true or false depending on the conditions, which controls whether the MMI tests will proceed.

Benefits

Increased efficiency: MMI Playwright tests are run only when necessary, reducing CI resource usage and speeding up the overall workflow.
Clearer conditions for MMI tests: The conditions for triggering MMI-specific tests are now based on both labels and reviewer assignments, ensuring greater flexibility and accuracy in when the tests are executed.

Related issues

Fixes:

Manual testing steps

Go to the CircleCI Project Page:

Navigate to the project pipeline in CircleCI.

Trigger a Workflow:

Manually trigger the workflow by running a new build. You can either rerun a previous build or push a new commit to the branch you’re testing.

Monitor the Workflow:

Observe the pipeline’s progress and ensure that the correct jobs (like MMI Playwright tests) are triggered based on your implemented conditions.
Check that the check-mmi-trigger step is evaluated correctly and logs the expected conditions met or skipped.

Verify Expected Behavior:

If the conditions for MMI tests (label/reviewer) are met, verify that the prep-build-test-mmi-playwright and test-e2e-mmi-playwright steps run successfully.
If conditions are not met, ensure the MMI tests are skipped as expected.

Check Logs:

Review the logs for each step to confirm that the correct conditions triggered the tests or that they were correctly skipped.

Screenshots/Recordings

Before

After

Pre-merge author checklist

I've followed MetaMask Contributor Docs and MetaMask Extension Coding Standards.
I've completed the PR template to the best of my ability
I’ve included tests if applicable
I’ve documented my code using JSDoc format if applicable
I’ve applied the right labels on the PR (see labeling guidelines). Not required for external contributors.

Pre-merge reviewer checklist

I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

github-actions · 2024-10-17T13:48:14Z

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

metamaskbot · 2024-10-17T14:25:21Z

Builds ready [fee43b2]

builds: chrome, firefox
builds (beta): chrome
builds (flask): chrome, firefox
builds (MMI): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
mv3: Background Module Init Stats
mv3: UI Init Stats
mv3: Module Load Stats
mv3: Bundle Size Stats
mv2: E2e Actions Stats
code coverage: Report
storybook: Storybook
typescript migration: Dashboard
all artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5, 6
- common: 0, 1, 10, 11, 2, 3, 4, 5, 6, 7, 8, 9
- content-script: 0
- offscreen: 0
- ui: 0, 1, 10, 11, 12, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (2103 ± 212 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	1651	3422	2091	406	195
		domContentLoaded	1599	2640	2011	300	144
		load	1645	3615	2103	441	212
		domInteractive	13	236	62	58	28
		backgroundConnect	9	919	88	196	94
		firstReactRender	47	188	103	28	13
		getState	4	115	28	33	16
		initialActions	0	1	0	0	0
		loadScripts	1189	2117	1515	242	116
		setupStore	11	90	30	23	11
		uiStartup	1857	3889	2348	494	237

Bundle size diffs

background: 0 Bytes (0.00%)
ui: 0 Bytes (0.00%)
common: 0 Bytes (0.00%)

metamaskbot · 2024-10-17T17:18:37Z

Builds ready [c135693]

builds: chrome, firefox
builds (beta): chrome
builds (flask): chrome, firefox
builds (MMI): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
mv3: Background Module Init Stats
mv3: UI Init Stats
mv3: Module Load Stats
mv3: Bundle Size Stats
mv2: E2e Actions Stats
code coverage: Report
storybook: Storybook
typescript migration: Dashboard
all artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5, 6
- common: 0, 1, 10, 11, 2, 3, 4, 5, 6, 7, 8, 9
- content-script: 0
- offscreen: 0
- ui: 0, 1, 10, 11, 12, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (2101 ± 169 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	252	2859	2013	532	255
		domContentLoaded	1579	2796	2056	331	159
		load	1590	2856	2101	352	169
		domInteractive	25	163	60	32	15
		backgroundConnect	9	132	37	34	16
		firstReactRender	43	304	100	52	25
		getState	5	92	28	24	12
		initialActions	0	0	0	0	0
		loadScripts	1153	2222	1535	288	138
		setupStore	10	101	39	27	13
		uiStartup	1775	3214	2339	422	203

Bundle size diffs

background: 0 Bytes (0.00%)
ui: 0 Bytes (0.00%)
common: 0 Bytes (0.00%)

metamaskbot · 2024-10-18T09:17:34Z

Builds ready [fb05163]

builds: chrome, firefox
builds (beta): chrome
builds (flask): chrome, firefox
builds (MMI): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
mv3: Background Module Init Stats
mv3: UI Init Stats
mv3: Module Load Stats
mv3: Bundle Size Stats
mv2: E2e Actions Stats
code coverage: Report
storybook: Storybook
typescript migration: Dashboard
all artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5, 6
- common: 0, 1, 10, 11, 2, 3, 4, 5, 6, 7, 8, 9
- content-script: 0
- offscreen: 0
- ui: 0, 1, 10, 11, 12, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (2115 ± 103 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	391	2779	2014	430	206
		domContentLoaded	1709	2521	2077	200	96
		load	1772	2694	2115	214	103
		domInteractive	25	104	50	19	9
		backgroundConnect	8	255	43	53	25
		firstReactRender	60	199	112	28	14
		getState	6	129	25	29	14
		initialActions	0	1	0	0	0
		loadScripts	1245	1865	1525	165	79
		setupStore	13	92	40	27	13
		uiStartup	1976	3424	2375	317	152

Bundle size diffs

background: 0 Bytes (0.00%)
ui: 0 Bytes (0.00%)
common: 0 Bytes (0.00%)

metamaskbot · 2024-10-18T13:08:02Z

Builds ready [faf540d]

builds: chrome, firefox
builds (beta): chrome
builds (flask): chrome, firefox
builds (MMI): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
mv3: Background Module Init Stats
mv3: UI Init Stats
mv3: Module Load Stats
mv3: Bundle Size Stats
mv2: E2e Actions Stats
code coverage: Report
storybook: Storybook
typescript migration: Dashboard
all artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5, 6
- common: 0, 1, 10, 11, 2, 3, 4, 5, 6, 7, 8, 9
- content-script: 0
- offscreen: 0
- ui: 0, 1, 10, 11, 12, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (1911 ± 89 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	291	2296	1735	496	238
		domContentLoaded	1636	2355	1862	180	87
		load	1640	2364	1911	185	89
		domInteractive	17	80	47	18	9
		backgroundConnect	9	146	53	37	18
		firstReactRender	49	285	94	51	24
		getState	5	179	37	41	20
		initialActions	0	1	0	0	0
		loadScripts	1159	1768	1351	138	66
		setupStore	11	95	36	25	12
		uiStartup	1795	2758	2166	257	123

Bundle size diffs

background: 0 Bytes (0.00%)
ui: 0 Bytes (0.00%)
common: 0 Bytes (0.00%)

albertolive · 2024-10-18T13:51:25Z

@Gudahtt could you change the ci/circleci: check-pr-tag to be check-mmi-trigger? It is expecting the old one

Gudahtt · 2024-10-21T13:09:44Z

.circleci/config.yml

+      - run:
+          name: Check for MMI Team Label or Reviewer
+          command: |
+            chmod +x .circleci/scripts/check_mmi_trigger.sh


The file permissions are checked into the repository; you don't need to make it executable every run, you can make it executable once and commit it.

Gudahtt · 2024-10-21T13:11:08Z

@albertolive Thanks for the ping. I have removed check-pr-tag as a required check. I'll leave the new one as optional for now as well

.circleci/scripts/check_mmi_trigger.sh

Gudahtt

Mostly looks good! We can remove the chmod +x step, and we still need to better handle PRs from forks

- Update check_mmi_trigger.sh to skip tests for forks without failing - Remove redundant chmod from CircleCI config

albertolive · 2024-10-22T07:59:39Z

@Gudahtt Thank you for your comments. I’ve updated the files accordingly. Regarding the tests not being mandatory, let's reconsider. Over the last few months, we’ve significantly improved the flaky tests, and making them mandatory would ensure greater confidence that nothing breaks moving forward. 🙏

cc @zone-live @shane-t

Requested changes have been made

metamaskbot · 2024-10-23T13:47:26Z

Builds ready [e262903]

builds: chrome, firefox
builds (beta): chrome
builds (flask): chrome, firefox
builds (MMI): chrome, firefox
builds (test): chrome, firefox
builds (test-flask): chrome, firefox
build viz: Build System
mv3: Background Module Init Stats
mv3: UI Init Stats
mv3: Module Load Stats
mv3: Bundle Size Stats
mv2: E2e Actions Stats
code coverage: Report
storybook: Storybook
typescript migration: Dashboard
all artifacts
bundle viz:
- background: 0, 1, 2, 3, 4, 5, 6
- common: 0, 1, 10, 11, 2, 3, 4, 5, 6, 7, 8, 9
- content-script: 0
- offscreen: 0
- ui: 0, 1, 10, 11, 12, 2, 3, 4, 5, 6, 7, 8, 9

Page Load Metrics (1935 ± 88 ms)

Platform	Page	Metric	Min (ms)	Max (ms)	Average (ms)	StandardDeviation (ms)	MarginOfError (ms)
Chrome	Home	firstPaint	307	2490	1865	398	191
		domContentLoaded	1675	2403	1894	174	84
		load	1687	2491	1935	183	88
		domInteractive	20	69	40	14	7
		backgroundConnect	10	149	46	35	17
		firstReactRender	52	206	92	43	21
		getState	5	63	23	19	9
		initialActions	0	0	0	0	0
		loadScripts	1213	1955	1419	171	82
		setupStore	12	98	33	23	11
		uiStartup	1891	2657	2170	209	100

Bundle size diffs

background: 0 Bytes (0.00%)
ui: 0 Bytes (0.00%)
common: 0 Bytes (0.00%)

Gudahtt

LGTM!

feat: improved way to trigger mmi e2e tests

3211a46

github-actions bot added the team-mmi PRs from the MMI team label Oct 17, 2024

chore: wip

fee43b2

albertolive added 4 commits October 17, 2024 18:10

chore: wip

f072290

chore: wip

0a2d158

chore: wip

137f744

chore: wip

c135693

albertolive added 13 commits October 18, 2024 09:17

chore: fix

08ffde2

chore: wip

1f24442

chore: wip

a950217

chore: wip

2ef232c

chore: wip

a16b52f

chore: wip

450ac7b

chore: wip

5aa9989

test

9030d9b

test

4a1212b

test

9d12890

test

784d9f1

test

b589bad

test

fb05163

albertolive added 5 commits October 18, 2024 11:58

test

d07712f

test

ba6e617

test

2eeeb2b

test

31f32de

test

d6676f1

albertolive marked this pull request as ready for review October 18, 2024 10:21

chore: reverted code

faf540d

albertolive enabled auto-merge October 18, 2024 12:38

shane-t previously approved these changes Oct 18, 2024

View reviewed changes

zone-live previously approved these changes Oct 18, 2024

View reviewed changes

Gudahtt reviewed Oct 21, 2024

View reviewed changes

.circleci/scripts/check_mmi_trigger.sh Show resolved Hide resolved

Gudahtt previously requested changes Oct 21, 2024

View reviewed changes

fix: Improve MMI test trigger handling in CI

8844e35

- Update check_mmi_trigger.sh to skip tests for forks without failing - Remove redundant chmod from CircleCI config

albertolive dismissed stale reviews from zone-live and shane-t via 8844e35 October 22, 2024 07:49

Merge branch 'develop' into improve-mmi-e2e-trigger

5a54879

Merge branch 'develop' into improve-mmi-e2e-trigger

e262903

albertolive requested review from zone-live, Gudahtt and shane-t October 23, 2024 13:35

zone-live approved these changes Oct 24, 2024

View reviewed changes

DDDDDanica approved these changes Oct 24, 2024

View reviewed changes

Gudahtt approved these changes Oct 25, 2024

View reviewed changes

albertolive added this pull request to the merge queue Oct 25, 2024

Merged via the queue into develop with commit d7d3b6d Oct 25, 2024
76 checks passed

albertolive deleted the improve-mmi-e2e-trigger branch October 25, 2024 21:22

github-actions bot locked and limited conversation to collaborators Oct 25, 2024

metamaskbot added the release-12.7.0 Issue or pull request that will be included in release 12.7.0 label Oct 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: improved way to trigger mmi e2e tests #27932

feat: improved way to trigger mmi e2e tests #27932

albertolive commented Oct 17, 2024 •

edited

Loading

github-actions bot commented Oct 17, 2024

metamaskbot commented Oct 17, 2024

metamaskbot commented Oct 17, 2024

metamaskbot commented Oct 18, 2024

metamaskbot commented Oct 18, 2024

albertolive commented Oct 18, 2024

Gudahtt Oct 21, 2024

Gudahtt commented Oct 21, 2024

Gudahtt left a comment

albertolive commented Oct 22, 2024

metamaskbot commented Oct 23, 2024

Gudahtt left a comment

feat: improved way to trigger mmi e2e tests #27932

feat: improved way to trigger mmi e2e tests #27932

Conversation

albertolive commented Oct 17, 2024 • edited Loading

Description

Related issues

Manual testing steps

Screenshots/Recordings

Before

After

Pre-merge author checklist

Pre-merge reviewer checklist

github-actions bot commented Oct 17, 2024

metamaskbot commented Oct 17, 2024

metamaskbot commented Oct 17, 2024

metamaskbot commented Oct 18, 2024

metamaskbot commented Oct 18, 2024

albertolive commented Oct 18, 2024

Gudahtt Oct 21, 2024

Choose a reason for hiding this comment

Gudahtt commented Oct 21, 2024

Gudahtt left a comment

Choose a reason for hiding this comment

albertolive commented Oct 22, 2024

metamaskbot commented Oct 23, 2024

Gudahtt left a comment

Choose a reason for hiding this comment

albertolive commented Oct 17, 2024 •

edited

Loading