Core performance tests improvements

[swissspidy]

Split out from #1093 to focus on solely the improvements to core/GB performance tests

• Improve demo content
  • Needs an owner
  • Priority?
• Add specific use cases (e.g., homepage, page with large LCP image, etc)
  • Also relevant for things like interactivity and layout stability metrics
  • Needs some fleshing out
  • Priority?
• Better dashboard with raw data
  • Needs an owner
  • Priority?
• Add tests for Twenty Twenty-Four
  • This was added in one of the recent refactorings. 2021, 2023, and 2024 are now tested.
• Alignment between core and Gutenberg
  • Core relies on GitHub Actions, while GB uses a sophisticated node script for running tests.
  • Should we have 1 script for both that could also be used by the ecosystem via npm or similar? Or is a GitHub Action good enough?

[adamsilverstein]

what about running the comparison checks between trunk and the previous release (with compare-wp-performance) on a daily basis (rather than tied to a commit) and plotting the p50 differentials (rather than plotting raw numbers)?

This is the check I ran at the end of the release cycle which showed a regression so I feel like running these daily could be useful.

[swissspidy]

Aren't the core performance tests and https://www.codevitals.run/project/wordpress supposed to help with that? Sounds like we'd be duplicating efforts.

[adamsilverstein]

Aren't the core performance tests and codevitals.run/project/wordpress supposed to help with that? Sounds like we'd be duplicating efforts.

Right, That dashboard is close to what I am suggestion. If those tests already compare against the previous release, it isn't clear from the dashboard, so maybe its more of a UI issue.

Ideally, I want to run the same tests at the release stage that we have been running all along with the action.

[swissspidy]

The core performance test workflow is callable, so we could add support for manually comparing latest vs. previous. That would kinda make https://github.com/swissspidy/compare-wp-performance obsolete.