-
Notifications
You must be signed in to change notification settings - Fork 245
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: compiler benchmarks and gh action #3503
Conversation
Adds a basic compiler benchmark suite and github action to track results over time.
Added DNM as benchmark.js doesn't really support what we want to do. Writing a basic custom runner that generates output that the action we use supports so we should still get the graphs on gh-pages and the comments for regression exceeding defined threshold. |
Changed strategy a bit since last review @RomainMuller @yuth
I also investigated including profiles and graphing difference in cpu/memory usage as well but it's non-trivial and we can add it after the fact if that data is interesting to us. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A bunch of cosmetic feedback, feel free to ship it after having reviewed that...
path.join(os.tmpdir(), 'jsii-cdk-bench-snapshot'), | ||
); | ||
cp.execSync(`tar xf ${cdkv2_21_1}`, { cwd: sourceDir }); | ||
cp.execSync('npm ci', { cwd: sourceDir }); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
npm ci
? Isn't CDK using yarn
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it is, but the snapshot we are taking is outside of the yarn workspace and we run npm install
. we can alternatively do yarn install and generate a yarn.lock but this assumes less about the environment?
Support async setup functions and use `tar` with a stream instead of shelling out to avoid potential cross-platform issues and OOM errors.
Thank you for contributing! ❤️ I will now look into making sure the PR is up-to-date, then proceed to try and merge it! |
Merging (with squash)... |
Adds a private package @jsii/benchmarks which includes a basic command line application for running jsii benchmarks.
Adds a custom benchmark runner leveraging the NodeJS
perf_hooks
module to time function calls across multiple iterations and return averaged results.Adds github action workflows to run benchmarks on PRs and compare performance to the target branch. This action will fail when a test suite is slower by 200%, but this threshold can be configured.
Adds to the gh-pages action workflow to run benchmarks and append new results to a file that is displayed as a graph on our docs site at the
/dev/bench
url. This will show benchmark suite results over time and allow us to track overall change in compiler performance across multiple commits.Example of benchmark results comment MrArnoldPalmer#417 (comment)
Example of benchmark results graph (but only 1 result cause just testing) https://mrarnoldpalmer.github.io/jsii/dev/bench/
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.