Skip to content

Commit

Permalink
feat: improved way to trigger mmi e2e tests (#27932)
Browse files Browse the repository at this point in the history
## **Description**

**Summary**

This pull request introduces a conditional check in the CircleCI
pipeline to determine whether MMI Playwright tests should be executed
based on the presence of the `team-mmi` label or if a reviewer from the
`MetaMask/mmi` team has been assigned to the pull request. This
optimizes the CI workflow by running MMI-specific tests only when
necessary.

Previously, the end-to-end tests were only triggered when the `team-mmi`
label was added to a pull request. However, in practice, no one outside
of our team was adding this label, which resulted in the end-to-end MMI
tests never being executed.

We already have an automation in place to automatically add a reviewer
from the `MetaMask/mmi` team when relevant pull requests are opened.
This pull request leverages that automation to ensure that the MMI tests
are automatically triggered when a reviewer is assigned, making the
process more efficient and reducing manual steps.

**Changes Implemented**

1. MMI Trigger Check

- Added a `check-mmi-trigger` step that runs a shell script to check if
the pull request has either:
  - A `team-mmi` label.
  - A reviewer from the `MetaMask/mmi` team assigned.
- If either condition is met, the MMI-specific tests will be triggered.
Otherwise, the workflow halts and skips these tests.

2. Modified Workflow

- The `prep-build-test-mmi-playwright` and `test-e2e-mmi-playwright`
steps are now conditionally executed based on the result of the
`check-mmi-trigger` step.
- If the MMI trigger is not present, the workflow halts after the
`check-mmi-trigger` step, improving pipeline efficiency by avoiding
unnecessary test execution.

3. Shell Script (check_mmi_trigger.sh)

- Added a script to check for the presence of the `team-mmi` label or
the `MetaMask/mmi` reviewer in a pull request.
- The script sets the `run_mmi_tests` environment variable to true or
false depending on the conditions, which controls whether the MMI tests
will proceed.

**Benefits**

- Increased efficiency: MMI Playwright tests are run only when
necessary, reducing CI resource usage and speeding up the overall
workflow.
- Clearer conditions for MMI tests: The conditions for triggering
MMI-specific tests are now based on both labels and reviewer
assignments, ensuring greater flexibility and accuracy in when the tests
are executed.


## **Related issues**

Fixes:

## **Manual testing steps**

1.	Go to the CircleCI Project Page:
- Navigate to the project pipeline in CircleCI.
2.	Trigger a Workflow:
- Manually trigger the workflow by running a new build. You can either
rerun a previous build or push a new commit to the branch you’re
testing.
3.	Monitor the Workflow:
- Observe the pipeline’s progress and ensure that the correct jobs (like
MMI Playwright tests) are triggered based on your implemented
conditions.
- Check that the `check-mmi-trigger` step is evaluated correctly and
logs the expected conditions met or skipped.
4.	Verify Expected Behavior:
- If the conditions for MMI tests (label/reviewer) are met, verify that
the `prep-build-test-mmi-playwright` and `test-e2e-mmi-playwright` steps
run successfully.
- If conditions are not met, ensure the MMI tests are skipped as
expected.
5.	Check Logs:
- Review the logs for each step to confirm that the correct conditions
triggered the tests or that they were correctly skipped.

## **Screenshots/Recordings**

<!-- If applicable, add screenshots and/or recordings to visualize the
before and after of your change. -->

### **Before**


![image](https://github.com/user-attachments/assets/fcf32cbd-f552-41f9-8a71-000554111aa6)


### **After**


![image](https://github.com/user-attachments/assets/d7470471-187a-4f28-90f3-d2e47d23f58c)

![image](https://github.com/user-attachments/assets/be5b1301-7204-4eaf-8cc3-381b990ddd21)


## **Pre-merge author checklist**

- [x] I've followed [MetaMask Contributor
Docs](https://github.com/MetaMask/contributor-docs) and [MetaMask
Extension Coding
Standards](https://github.com/MetaMask/metamask-extension/blob/develop/.github/guidelines/CODING_GUIDELINES.md).
- [x] I've completed the PR template to the best of my ability
- [x] I’ve included tests if applicable
- [x] I’ve documented my code using [JSDoc](https://jsdoc.app/) format
if applicable
- [x] I’ve applied the right labels on the PR (see [labeling
guidelines](https://github.com/MetaMask/metamask-extension/blob/develop/.github/guidelines/LABELING_GUIDELINES.md)).
Not required for external contributors.

## **Pre-merge reviewer checklist**

- [x] I've manually tested the PR (e.g. pull and build branch, run the
app, test code being changed).
- [x] I confirm that this PR addresses all acceptance criteria described
in the ticket it closes and includes the necessary testing evidence such
as recordings and or screenshots.
  • Loading branch information
albertolive authored Oct 25, 2024
1 parent 32a3b07 commit d7d3b6d
Show file tree
Hide file tree
Showing 2 changed files with 88 additions and 47 deletions.
75 changes: 28 additions & 47 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -87,15 +87,15 @@ aliases:
cat ${HOME}/project/.circleci/scripts/enable-vnc.sh >> ~/.bashrc
fi
# Check if MMI Optional tests should run
- &check-mmi-optional
name: Check if MMI Optional tests should run
# Check if MMI tests should run
- &check-mmi-trigger
name: Check if MMI tests should run
command: |
RUN_MMI_OPTIONAL=$(cat ./RUN_MMI_OPTIONAL)
if [[ "${RUN_MMI_OPTIONAL}" == "true" ]]; then
echo "Running MMI Optional tests"
source mmi_trigger.env
if [ "${run_mmi_tests}" == "true" ]; then
echo "Running MMI tests"
else
echo "Skipping MMI Optional tests"
echo "Skipping MMI tests"
circleci step halt
fi
Expand All @@ -114,7 +114,7 @@ workflows:
- trigger-beta-build:
requires:
- prep-deps
- check-pr-tag
- check-mmi-trigger
- prep-deps
- get-changed-files-with-git-diff:
filters:
Expand Down Expand Up @@ -179,7 +179,7 @@ workflows:
- prep-build-test-mmi-playwright:
requires:
- prep-deps
- check-pr-tag
- check-mmi-trigger
- prep-build-storybook:
requires:
- prep-deps
Expand Down Expand Up @@ -231,7 +231,7 @@ workflows:
requires:
- prep-build-test-mmi
- get-changed-files-with-git-diff
- test-e2e-mmi-playwright - OPTIONAL:
- test-e2e-mmi-playwright:
requires:
- prep-build-test-mmi-playwright
- test-e2e-chrome-rpc-mmi:
Expand Down Expand Up @@ -421,39 +421,6 @@ jobs:
name: Create GitHub Pull Request for version
command: .circleci/scripts/release-create-release-pr.sh

check-pr-tag:
docker:
- image: cimg/base:stable
steps:
- run:
name: Check for MMI Team Tag
command: |
#!/bin/bash
GH_LABEL=team-mmi
if [ -z "$CIRCLE_PULL_REQUESTS" ]; then
echo "Skipping tag check; this is not a PR."
echo "false" > ./RUN_MMI_OPTIONAL
exit 0
fi
echo $CIRCLE_PULL_REQUESTS | sed 's/,/\n/g'
# See if any associated PRs have matching label
HAS_MATCHING_PR=$(echo $CIRCLE_PULL_REQUESTS \
| sed -e 's#,#\n#g' -e 's#/github.com/#/api.github.com/repos/#g' -e 's#/pull/#/pulls/#g' \
| xargs -n1 curl -s \
| jq -s "map((.labels|map(select(.name==\"${GH_LABEL}\"))))|flatten|length > 0")
echo "${GH_LABEL} tag presence: ${HAS_MATCHING_PR}"
# assign the RUN_MMI_OPTIONAL variable
echo "${HAS_MATCHING_PR}" > ./RUN_MMI_OPTIONAL
- persist_to_workspace:
root: .
paths:
- RUN_MMI_OPTIONAL

prep-deps:
executor: node-browsers-medium
steps:
Expand Down Expand Up @@ -839,7 +806,7 @@ jobs:
- run: corepack enable
- attach_workspace:
at: .
- run: *check-mmi-optional
- run: *check-mmi-trigger
- run:
name: Build MMI extension for Playwright e2e
command: |
Expand All @@ -854,7 +821,6 @@ jobs:
- persist_to_workspace:
root: .
paths:
- RUN_MMI_OPTIONAL
- dist-test-mmi-playwright
- builds-test-mmi-playwright
- store_artifacts:
Expand Down Expand Up @@ -1306,15 +1272,15 @@ jobs:
- store_test_results:
path: test/test-results/e2e

test-e2e-mmi-playwright - OPTIONAL:
test-e2e-mmi-playwright:
executor: playwright
parallelism: 2
steps:
- run: *shallow-git-clone-and-enable-vnc
- run: corepack enable
- attach_workspace:
at: .
- run: *check-mmi-optional
- run: *check-mmi-trigger
- run:
name: Move test build to dist
command: mv ./dist-test-mmi-playwright ./dist
Expand Down Expand Up @@ -1743,3 +1709,18 @@ jobs:
- run:
name: All Tests Passed
command: echo 'whew - everything passed!'

check-mmi-trigger:
executor: node-browsers-medium
steps:
- checkout
- run:
name: Check for MMI Team Label or Reviewer
command: ./.circleci/scripts/check_mmi_trigger.sh
- store_artifacts:
path: mmi_trigger.env
destination: mmi_trigger.env
- persist_to_workspace:
root: .
paths:
- mmi_trigger.env
60 changes: 60 additions & 0 deletions .circleci/scripts/check_mmi_trigger.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#!/bin/bash
set -eo pipefail

# Ensure required environment variables are set
if [ -z "$CIRCLE_PULL_REQUEST" ] || [ -z "$GITHUB_TOKEN" ]; then
echo "This appears to be a fork or required environment variables are not set."
echo "Skipping MMI tests."
echo "run_mmi_tests=false" > mmi_trigger.env
exit 0
fi

# Extract PR number from the pull request URL
PR_NUMBER=$(echo "$CIRCLE_PULL_REQUEST" | awk -F'/' '{print $NF}')

# Define repository details
REPO_OWNER="$CIRCLE_PROJECT_USERNAME"
REPO_NAME=$(basename "$CIRCLE_REPOSITORY_URL" .git)

# Fetch PR details using GitHub API
PR_DETAILS=$(curl -s -H "Authorization: token $GITHUB_TOKEN" \
"https://api.github.com/repos/$REPO_OWNER/$REPO_NAME/pulls/$PR_NUMBER")

# Fetch submitted reviews
SUBMITTED_REVIEWS=$(curl -s -H "Authorization: token $GITHUB_TOKEN" \
"https://api.github.com/repos/$REPO_OWNER/$REPO_NAME/pulls/$PR_NUMBER/reviews")

# Check for label 'team-mmi'
LABEL_EXISTS=$(jq -r '.labels[]? | select(.name == "team-mmi") | length > 0' <<< "$PR_DETAILS")

# Check for individual reviewer 'mmi'
REVIEWER_REQUESTED=$(jq -r '.requested_reviewers[]? | select(.login == "mmi") | length > 0' <<< "$PR_DETAILS")

# Check for team reviewer 'mmi'
TEAM_REQUESTED=$(jq -r '.requested_teams[]? | select(.slug == "mmi") | length > 0' <<< "$PR_DETAILS")

# Check if 'mmi' submitted a review
REVIEWER_SUBMITTED=$(jq -r '.[]? | select(.user.login == "mmi") | length > 0' <<< "$SUBMITTED_REVIEWS")

# Determine which condition was met and trigger tests if needed
if [[ "$LABEL_EXISTS" == "true" || "$REVIEWER_REQUESTED" == "true" || "$TEAM_REQUESTED" == "true" || "$REVIEWER_SUBMITTED" == "true" ]]; then
echo "run_mmi_tests=true" > mmi_trigger.env

# Log exactly which condition was met
echo "Conditions met:"
if [[ "$LABEL_EXISTS" == "true" ]]; then
echo "- Label 'team-mmi' found."
fi
if [[ "$REVIEWER_REQUESTED" == "true" ]]; then
echo "- Reviewer 'mmi' requested."
fi
if [[ "$TEAM_REQUESTED" == "true" ]]; then
echo "- Team 'mmi' requested."
fi
if [[ "$REVIEWER_SUBMITTED" == "true" ]]; then
echo "- Reviewer 'mmi' submitted a review."
fi
else
echo "run_mmi_tests=false" > mmi_trigger.env
echo "Skipping MMI tests: Neither the 'team-mmi' label was found nor a reviewer from the 'MetaMask/mmi' team was assigned."
fi

0 comments on commit d7d3b6d

Please sign in to comment.