Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automated performance testing #21738

Open
Tracked by #246
novocaine opened this issue Apr 11, 2022 · 2 comments · Fixed by matrix-org/matrix-react-sdk#8586
Open
Tracked by #246

Automated performance testing #21738

novocaine opened this issue Apr 11, 2022 · 2 comments · Fixed by matrix-org/matrix-react-sdk#8586
Labels
A-Testing Testing, code coverage, etc. T-Enhancement

Comments

@novocaine
Copy link
Contributor

Your use case

What would you like to do?

Run performance tests against key functionality of the application (e.g. rendering, room switching) as part of CI.

Why would you like to do it?

Element has frequent performance regressions. We should guard against this to prevent it occurring; its often only detected a while later and its really difficult to tell which change caused the regression

How would you like to achieve it?

We could do this in Cypress by simulating scenarios that cause perf issues e.g. many rooms, many events in rooms etc.

Have you considered any alternatives?

No

Additional context

No response
@novocaine novocaine added T-Enhancement A-Testing Testing, code coverage, etc. labels Apr 11, 2022
@jryans jryans self-assigned this Apr 19, 2022
@novocaine novocaine mentioned this issue Apr 28, 2022
Closed
jryans added a commit to matrix-org/matrix-react-sdk that referenced this issue May 13, 2022
@jryans
Add basic performance testing via Cypress
f89f6a2 
This adds a basic approach to performance testing within Cypress tests. You
sprinkle `cy.startMeasuring` / `cy.stopMeasuring` pairs in the areas of
interest. The measurements are added to the Cypress log shown in interactive
runs and they are also assembled into a dashboard
(https://matrix-org.github.io/matrix-react-sdk/cypress/bench/) for tracking
changes across time.

Fixes element-hq/element-web#21738
jryans added a commit to matrix-org/matrix-react-sdk that referenced this issue May 13, 2022
@jryans
Add basic performance testing via Cypress
e75f4b3 
This adds a basic approach to performance testing within Cypress tests. You
sprinkle `cy.startMeasuring` / `cy.stopMeasuring` pairs in the areas of
interest. The measurements are added to the Cypress log shown in interactive
runs and they are also assembled into a dashboard
(https://matrix-org.github.io/matrix-react-sdk/cypress/bench/) for tracking
changes across time.

Fixes element-hq/element-web#21738
@jryans jryans mentioned this issue May 13, 2022
Merged
@novocaine novocaine reopened this May 19, 2022
@novocaine
Copy link
Contributor Author

novocaine commented May 19, 2022 •
edited
Loading

Tests are running with a lot of variance in time so we can't really leverage them just yet, so reopening.

@bkil bkil mentioned this issue Sep 14, 2022
Open
@Johennes
Copy link
Contributor

Johennes commented Feb 3, 2023

It looks like https://docs.cypress.io/faq/questions/using-cypress-faq#Can-Cypress-be-used-for-performance-testing cautions against using Cypress for performance testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-Testing Testing, code coverage, etc. T-Enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add basic performance testing via Cypress matrix-org/matrix-react-sdk
3 participants
@novocaine @jryans @Johennes