Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix "Test Snap Cronjob can trigger a cronjob to open a di..." flaky tests #28154

Closed
hjetpoluru opened this issue Oct 29, 2024 · 1 comment
Closed
Assignees
Labels
flaky tests Sev2-normal Normal severity; minor loss of service or inconvenience. team-extension-platform team-snaps-platform Snaps Platform team type-bug

Comments

@hjetpoluru
Copy link
Contributor

hjetpoluru commented Oct 29, 2024

CI failure reported on Oct 31 (Test Snap Cronjob can trigger a cronjob to open a di...)

Build - Flask
Browser - Chrome
Test Path - /test/e2e/snaps/test-snap-cronjob.spec.js
Test Scenario - Test Snap Cronjob can trigger a cronjob to open a dialog every minute

Error: No window found by background script with title: MetaMask Dialog
    at ServerMochaToBackground.receivedMessage (test/e2e/background-socket/server-mocha-to-background.ts:2:2485)
    at ws.onmessage (test/e2e/background-socket/server-mocha-to-background.ts:2:1916)
    at callListener (node_modules/ws/lib/event-target.js:290:14)
    at WebSocket.onMessage (node_modules/ws/lib/event-target.js:209:9)
    at WebSocket.emit (node:events:519:28)
    at WebSocket.emit (node:domain:488:12)
    at Receiver.receiverOnMessage (node_modules/ws/lib/websocket.js:1220:20)
    at Receiver.emit (node:events:519:28)
    at Receiver.emit (node:domain:488:12)
    at Receiver.dataMessage (node_modules/ws/lib/receiver.js:596:14)
    at Receiver.getData (node_modules/ws/lib/receiver.js:496:10)
    at Receiver.startLoop (node_modules/ws/lib/receiver.js:167:16)
    at Receiver._write (node_modules/ws/lib/receiver.js:94:10)
    at writeOrBuffer (node:internal/streams/writable:570:12)
    at _write (node:internal/streams/writable:499:10)
    at Writable.write (node:internal/streams/writable:508:10)
    at Socket.socketOnData (node_modules/ws/lib/websocket.js:1355:35)
    at Socket.emit (node:events:519:28)
    at Socket.emit (node:domain:488:12)
    at addChunk (node:internal/streams/readable:559:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:510:3)
    at Readable.push (node:internal/streams/readable:390:5)
    at TCP.onStreamRead (node:internal/stream_base_commons:191:23)

https://app.circleci.com/pipelines/github/MetaMask/metamask-extension/108186/workflows/1dc73b8c-a92d-4e87-a378-39d8db2563e2/jobs/4046081/tests#failed-test-0

Build - Flask
Browser - Firefox
Test Path - /test/e2e/snaps/test-snap-cronjob.spec.js
Test Scenario - Test Snap Cronjob can trigger a cronjob to open a dialog every minute

ElementNotInteractableError: Element <button id="connectcronjobs" class="btn btn-primary" type="submit"> could not be scrolled into view
  (Ran on CircleCI Node 9 of 24, Job test-e2e-firefox)
    at Object.throwDecodedError (node_modules/selenium-webdriver/lib/error.js:524:15)
    at parseHttpResponse (node_modules/selenium-webdriver/lib/http.js:601:13)
    at Executor.execute (node_modules/selenium-webdriver/lib/http.js:529:28)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async thenableWebDriverProxy.execute (node_modules/selenium-webdriver/lib/webdriver.js:745:17)
    at async element.click (test/e2e/webdriver/driver.js:79:7)
    at async Driver.clickElement (test/e2e/webdriver/driver.js:579:9)
    at async /home/circleci/project/test/e2e/snaps/test-snap-cronjob.spec.js:34:9
    at async withFixtures (test/e2e/helpers.js:217:5)
    at async Context.<anonymous> (test/e2e/snaps/test-snap-cronjob.spec.js:13:5)

image

image

@hjetpoluru hjetpoluru added type-bug Sev2-normal Normal severity; minor loss of service or inconvenience. flaky tests team-snaps-platform Snaps Platform team labels Oct 29, 2024
@github-project-automation github-project-automation bot moved this to To be fixed in Bugs by team Oct 29, 2024
@github-project-automation github-project-automation bot moved this to To be fixed in Bugs by severity Oct 29, 2024
@hjetpoluru
Copy link
Contributor Author

The flakiness initially reported in the ticket, has been fixed in the PR #27684.

But when this PR #28153 was unable to be merged, a discussion took place in the slack channel and this flaky test appeared with another reason details in the CI link.

According to the log, the failure occurs in the Chrome browser when the delay await driver.delayFirefox(1000); is executed, which should not have been executed. Hence further analysis of the flaky test needs to be done.

@hjetpoluru hjetpoluru self-assigned this Nov 4, 2024
github-merge-queue bot pushed a commit that referenced this issue Nov 12, 2024
…tests (#28363)

<!--
Please submit this PR as a draft initially.
Do not mark it as "Ready for review" until the template has been
completely filled out, and PR status checks have passed at least once.
-->

## **Description**

This flaky test is due to the dialog taking time to appear and
disappear, which is the expected behavior. I have attached a video
demonstrating the current behavior.


https://github.com/user-attachments/assets/66c68d05-a2dc-4bb5-8870-c967ea9d630a

To address this issue, I added retry logic and validation. However, I
noticed that when the dialog does not appear, `switchToWindowWithTitle`
method throws an exception and fails the test hence causing the retry
logic to be ineffective. Implementation is in the first commit in this
PR b97dbbe.

Fixing this issue within the framework would require significant effort
and could impact other tests. Therefore, I have implemented a wait
period before attempting to switch to the window. This approach ensures
that the dialog has enough time to appear, thereby reducing the
likelihood of test failures.

<!--
Write a short description of the changes included in this pull request,
also include relevant motivation and context. Have in mind the following
questions:
1. What is the reason for the change?
2. What is the improvement/solution?
-->

[![Open in GitHub
Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/MetaMask/metamask-extension/pull/28363?quickstart=1)

## **Related issues**

Fixes:
#28154

## **Manual testing steps**

Run the test locally or in codespaces using below command
yarn
yarn build:test:webpack
ENABLE_MV3=false yarn test:e2e:single
test/e2e/snaps/test-snap-cronjob.spec.js --browser=chrome
--leave-running

## **Pre-merge author checklist**

- [x] I've followed [MetaMask Contributor
Docs](https://github.com/MetaMask/contributor-docs) and [MetaMask
Extension Coding
Standards](https://github.com/MetaMask/metamask-extension/blob/develop/.github/guidelines/CODING_GUIDELINES.md).
- [x] I've completed the PR template to the best of my ability
- [x] I’ve included tests if applicable
- [ ] I’ve documented my code using [JSDoc](https://jsdoc.app/) format
if applicable
- [x] I’ve applied the right labels on the PR (see [labeling
guidelines](https://github.com/MetaMask/metamask-extension/blob/develop/.github/guidelines/LABELING_GUIDELINES.md)).
Not required for external contributors.

## **Pre-merge reviewer checklist**

- [ ] I've manually tested the PR (e.g. pull and build branch, run the
app, test code being changed).
- [ ] I confirm that this PR addresses all acceptance criteria described
in the ticket it closes and includes the necessary testing evidence such
as recordings and or screenshots.
@github-project-automation github-project-automation bot moved this from To be fixed to Fixed in Bugs by severity Nov 12, 2024
@github-project-automation github-project-automation bot moved this from To be fixed to Fixed in Bugs by team Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flaky tests Sev2-normal Normal severity; minor loss of service or inconvenience. team-extension-platform team-snaps-platform Snaps Platform team type-bug
Projects
Archived in project
Development

No branches or pull requests

2 participants