Make the performance tests more stable #48094
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
youknowriad
added
the
[Type] Build Tooling
|
Size Change: 0 B Total Size: 1.33 MB ℹ️ View Unchanged
|
|
Flaky tests detected in 67998fb. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/4231927585
|
WunderBart
reviewed
Feb 16, 2023
|
I've noticed a typo in the selector and fixed it. Let's see if it helps. |
|
Still broken, but now something else. It looks like the I've also noticed that the Site Editor's Typing suite is quite chaotic, as it doesn’t wait properly for the post content to be loaded and ends up typing the I'm currently trying to address both issues, will keep you updated. |
|
@youknowriad which editor canvas selectors should be supported in the performance tests? AFAIK there's the |
|
I can't think of any other canvas selector. Also this is broken in trunk which is basically the latest version. |
|
Pushed a WiP fix here: #48138. Will continue tomorrow. |
youknowriad
force-pushed
the
test/stabilize-perf-tests
branch
from
dfbaa2c
to
468711e
Compare
youknowriad
force-pushed
the
test/stabilize-perf-tests
branch
from
d23fdb9
to
67998fb
Compare
youknowriad
added
the
Backport to WP 6.7 Beta/RC
youknowriad
commented
Feb 21, 2023
| // where the site editor toggle was not implemented yet. | ||
| if ( ! isViewMode ) { | ||
| return; | ||
| try { |
There was a problem hiding this comment.
The change is not necessary but I think it's a bit safer than the previous code, so I decided to leave it.
youknowriad
commented
Feb 21, 2023
| 'npm ci && npm run prebuild:packages && node ./bin/packages/build.js && npx wp-scripts build', | ||
| buildPath | ||
| ); | ||
|
|
There was a problem hiding this comment.
In #45737 we've introduced a potential optimization to the performance job, the problem is that it resulted in the job comparing the wrong thing.
Basically if a branch was equivalent to the "test branch", we reused the same code to gain some time and avoid the git clone, the issue is that we didn't build the code in that case, meaning we were using the WP trunk version of Gutenberg and not the actual branch.
(Currently WP trunk's version does have the '/navigation/single' support in the site editor which explains why this random change fixed the perf job.
The bummer is that you can say that for the last three months, the perf jobs in PRs were useless.
There was a problem hiding this comment.
That also explains why I was completely unable to repro locally. 💥
Nice catch, though the bug is indeed a nasty one! 🐛
ellatrix
approved these changes
Feb 21, 2023
ntsekouras
pushed a commit
that referenced
this pull request
Feb 27, 2023
|
I just cherry-picked this PR to the wp/6.2 branch to get it included in the next release: 6771f8b |
ntsekouras
removed
the
Backport to WP 6.7 Beta/RC
WunderBart
added a commit
that referenced
this pull request
Feb 28, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Recently, the performance job has been unstable. This PR tries to solve that.
It turns out the job was broken for a long time. Read this comment for the details.