Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky playwright test: crypto/event-shields.spec.ts: should show correct shields on events sent by devices which have since been deleted #28061

Closed
RiotRobot opened this issue Sep 17, 2024 · 59 comments · Fixed by element-hq/matrix-react-sdk#143 or #28641
Closed

Flaky playwright test: crypto/event-shields.spec.ts: should show correct shields on events sent by devices which have since been deleted #28061

RiotRobot opened this issue Sep 17, 2024 · 59 comments · Fixed by element-hq/matrix-react-sdk#143 or #28641
Assignees
@richvdh
Labels
A-E2EE Z-Flaky-Test A test is raising false alarms

Comments

@RiotRobot
Copy link
Contributor

https://github.com/element-hq/matrix-react-sdk/actions/runs/10908576705
@RiotRobot RiotRobot added the Z-Flaky-Test A test is raising false alarms label Sep 17, 2024
@dosubot dosubot bot added the A-E2EE label Sep 17, 2024
@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/10919174293

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/10919624690

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/10937689615

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/10938679884

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/10944560453

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/10992656361

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/11034447584

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/11054600607

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/11108175970

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/11110803603

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/11110825464

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/11125667597

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/11175849515

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/11232565367

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/11233921980

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/11235210852

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/11257608084

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/11264342274

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/11269690192

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/11276085267

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/11276849979

@dbkr
Copy link
Member

dbkr commented Oct 10, 2024

This has flaked twice in a row for me. If nobody has time to look, I think we'll have to disable it.

@dbkr
Copy link
Member

dbkr commented Oct 10, 2024

ftr, this is:

https://e2e-131--matrix-react-sdk.netlify.app/#?testId=549c35dde3453c6f6198-53d1af2a964b4d8c9a63

Error: jsHandle.evaluate: Error: Cannot encrypt event in unconfigured room !wDXBgADKUDbRHIRBhX:localhost

    at RustCrypto.encryptEvent (http://localhost:8080/bundles/94c04cf2a65f14e9f347/771.js:3276:13)
    at MatrixClient.encryptEventIfNeeded (http://localhost:8080/bundles/94c04cf2a65f14e9f347/init.js:9339:30)
    at async MatrixClient.encryptAndSendEvent (http://localhost:8080/bundles/94c04cf2a65f14e9f347/init.js:9271:9)
    at async <anonymous>:260:30
    at RustCrypto.encryptEvent (/home/runner/work/matrix-react-sdk/matrix-react-sdk/matrix-react-sdk/http:/localhost:8080/bundles/94c04cf2a65f14e9f347/771.js:3276:13)
    at MatrixClient.encryptEventIfNeeded (/home/runner/work/matrix-react-sdk/matrix-react-sdk/matrix-react-sdk/http:/localhost:8080/bundles/94c04cf2a65f14e9f347/init.js:9339:30)
    at async MatrixClient.encryptAndSendEvent (/home/runner/work/matrix-react-sdk/matrix-react-sdk/matrix-react-sdk/http:/localhost:8080/bundles/94c04cf2a65f14e9f347/init.js:9271:9)
    at /home/runner/work/matrix-react-sdk/matrix-react-sdk/matrix-react-sdk/async <anonymous>:260:30
    at Bot.sendMessage (/home/runner/work/matrix-react-sdk/matrix-react-sdk/matrix-react-sdk/playwright/pages/client.ts:125:23)
    at /home/runner/work/matrix-react-sdk/matrix-react-sdk/matrix-react-sdk/playwright/e2e/crypto/event-shields.spec.ts:276:13

ie. the bot client is failing to send a message into the room because it's not ready yet, so probably needs more waiting on the bot side?

@richvdh richvdh mentioned this issue Oct 11, 2024
Merged
@RiotRobot RiotRobot reopened this Oct 16, 2024
@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/matrix-react-sdk/actions/runs/11360236274

@richvdh
Copy link
Member

richvdh commented Oct 16, 2024

ugh.

  3) crypto/event-shields.spec.ts:273:13 › Cryptography › event shields › should show correct shields on events sent by devices which have since been deleted 
    Error: Timed out 5000ms waiting for expect(locator).not.toBeVisible()

    Locator: locator('.mx_EventTile').filter({ hasText: 'test encrypted from verified' }).locator('.mx_EventTile_e2eIcon')
    Expected: not visible
    Received: visible
    Call log:
      - expect.not.toBeVisible with timeout 5000ms
      - waiting for locator('.mx_EventTile').filter({ hasText: 'test encrypted from verified' }).locator('.mx_EventTile_e2eIcon')
      -   locator resolved to <div tabindex="0" aria-labelledby="floating-ui-248" aria-label="State of the end-to-end encryption" class="mx_EventTile_e2eIcon mx_EventTile_e2eIcon_warning"></div>
      -   unexpected value "visible"
      -   locator resolved to <div tabindex="0" aria-labelledby="floating-ui-248" aria-label="State of the end-to-end encryption" class="mx_EventTile_e2eIcon mx_EventTile_e2eIcon_warning"></div>
      -   unexpected value "visible"
      -   locator resolved to <div tabindex="0" aria-labelledby="floating-ui-248" aria-label="State of the end-to-end encryption" class="mx_EventTile_e2eIcon mx_EventTile_e2eIcon_warning"></div>
      -   unexpected value "visible"
      -   locator resolved to <div tabindex="0" aria-labelledby="floating-ui-248" aria-label="State of the end-to-end encryption" class="mx_EventTile_e2eIcon mx_EventTile_e2eIcon_warning"></div>
      -   unexpected value "visible"
      -   locator resolved to <div tabindex="0" aria-labelledby="floating-ui-248" aria-label="State of the end-to-end encryption" class="mx_EventTile_e2eIcon mx_EventTile_e2eIcon_warning"></div>
      -   unexpected value "visible"
      -   locator resolved to <div tabindex="0" aria-labelledby="floating-ui-248" aria-label="State of the end-to-end encryption" class="mx_EventTile_e2eIcon mx_EventTile_e2eIcon_warning"></div>
      -   unexpected value "visible"
      -   locator resolved to <div tabindex="0" aria-labelledby="floating-ui-248" aria-label="State of the end-to-end encryption" class="mx_EventTile_e2eIcon mx_EventTile_e2eIcon_warning"></div>
      -   unexpected value "visible"
      -   locator resolved to <div tabindex="0" aria-labelledby="floating-ui-248" aria-label="State of the end-to-end encryption" class="mx_EventTile_e2eIcon mx_EventTile_e2eIcon_warning"></div>
      -   unexpected value "visible"
      -   locator resolved to <div tabindex="0" aria-labelledby="floating-ui-248" aria-label="State of the end-to-end encryption" class="mx_EventTile_e2eIcon mx_EventTile_e2eIcon_warning"></div>
      -   unexpected value "visible"


      306 |
      307 |             const penultimate = page.locator(".mx_EventTile").filter({ hasText: "test encrypted from verified" });
    > 308 |             await expect(penultimate.locator(".mx_EventTile_e2eIcon")).not.toBeVisible();
          |                                                                            ^
      309 |         });
      310 |     });
      311 | });

        at /home/runner/work/matrix-react-sdk/matrix-react-sdk/matrix-react-sdk/playwright/e2e/crypto/event-shields.spec.ts:308:76

It's a different failure mode to before.

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/11737389655

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/11739465398

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/11754520245

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/11801363669

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/11816958271

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/11853681474

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/11891060295

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/11938564633

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/11974264276

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/12028458391

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/12037105158

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/12037415719

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/12047113599

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/12047168330

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/12049793206

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/12052798693

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/12053637583

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/12055165906

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/12068815564

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/12116483314

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/12117246038

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/12117724470

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/12139583496

@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/12142011517

@richvdh richvdh self-assigned this Dec 4, 2024
@RiotRobot
Copy link
Contributor Author

https://github.com/element-hq/element-web/actions/runs/12157280053

richvdh added a commit that referenced this issue Dec 4, 2024
@richvdh
Playwright: fix (hopefully) flaky shields test
4607325 
Wait for our user to fetch the bot's identity before running the test, to work
around a race in the shield logic.

Hopefully, fixes #28061
@richvdh richvdh mentioned this issue Dec 4, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Dec 4, 2024
@richvdh
Playwright: fix (hopefully) flaky shields test (#28641)
5547101 
* Playwright: improve failure report when an unexpected shield exists

If we discover an E2E shield when we didn't expect one, let's make the error
message more helpful by checking the tooltip.

* Playwright: fix (hopefully) flaky shields test

Wait for our user to fetch the bot's identity before running the test, to work
around a race in the shield logic.

Hopefully, fixes #28061
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@richvdh richvdh

Labels
A-E2EE Z-Flaky-Test A test is raising false alarms
Projects
None yet
Milestone
No milestone
3 participants
@dbkr @richvdh @RiotRobot