Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resume sending E2E test failures to slack #3575

Merged
merged 4 commits into from
Dec 24, 2021

Conversation

deepakpathania
Copy link
Contributor

@deepakpathania deepakpathania commented Dec 24, 2021

Fixes #3555

Changes proposed in this Pull Request

  • Add the required config for using e2e-environment to resume sending E2E test failures to slack.
  • Rename jest-setup.js to jest.setup.js and move it to tests/e2e/config for it to be picked up correctly.
  • Use useE2EJestConfig for jest config to use the failure script that includes slack reporting.

Testing instructions

  • Go to Actions tab -> WC Pay E2E tests.
  • Run the workflow for this branch update/slack-e2e-reporting.
  • Notice failure messages would be sent to #wcpay-bots channel in slack, reference job.

Sample slack screenshot

Screenshot 2021-12-24 at 1 12 28 PM


  • Added changelog entries to both readme.txt and changelog.txt (or does not apply)
  • Covered with tests (or have a good reason not to test in description ☝️)
  • Tested on mobile (or does not apply)

Post merge

achyuthajoy and others added 2 commits December 24, 2021 14:46
* Move user management functions to env setup

* Update user management script order + moved values to vars
Copy link
Contributor

@naman03malhotra naman03malhotra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, left a comment for my understanding.

Copy link
Contributor

@zmaglica zmaglica left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@deepakpathania deepakpathania merged commit c06e3ea into develop Dec 24, 2021
@deepakpathania deepakpathania deleted the fix/slack-e2e-reporting branch December 24, 2021 12:15
name: screenshots
path: screenshots
name: e2e-screenshots
path: tests/e2e/screenshots
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was there a change to where the screenshots get saved? It happens here but maybe when running locally it depends on where the tests are run from? Not seeing the artifact on recent runs (e.g. here).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dechov We moved to use the useE2EJestConfig provided by e2e-environment which sends the failed screenshot messages directly to #wcpay-bots channel in slack, which might have overridden the old artifact behavior to only create artifacts for failure cases, see this run for reference.
Do we need the screenshots for success scenarios as well?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see 👍

Not that we need the regular ones but there is some theoretical benefit to having snapshots from each distinct screen from the normal run, to ensure that there aren't visual regressions that wouldn't be caught by specific assertions. What do you think of having a "failure screenshots" artifact in addition to the earlier one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel unless there is an alert/checking process in place, we wouldn't really monitor the successful ones regularly, and manually checking visual regressions could be tricky across runs.
We could either ask GS to do some sanity or have explicit visual regression tests in place for the same but that's just my opinion, we could discuss it internally to see what other folks feel about the same.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Spike: report E2E test failures to slack
5 participants