Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reports we should generate for periodic benchmark jobs #5

Open
jrevels opened this issue Jun 11, 2016 · 1 comment
Open

Reports we should generate for periodic benchmark jobs #5

jrevels opened this issue Jun 11, 2016 · 1 comment

Comments

@jrevels
Copy link
Member

jrevels commented Jun 11, 2016

From discussion here:

Does it make sense to run against previous release as well. I'm thinking else, regressions could be introduced bit by bit were each part is small enough to dissappear in the noise.

After we gather enough nightly data, we'll be able to start looking at trends over longer time intervals, which helps with this. My idea was to have @nanosoldier generate a weekly/monthly report using the existing benchmark data, in addition to the daily reports. Then we'd have daily comparisons, weekly comparisons, and monthly comparisons, as well as the data to do more advanced trend analysis later on.

We can simply run the benchmarks for the current stable release once, and reuse that data when doing comparisons. Then we could just generate report_vs_release.md in addition to our current report.md.

@KristofferC
Copy link
Contributor

Just a small comment. Does it make sense to put all the daily reports under a common daily folder in BaseBenchmarkReports? I know that using Github as data storage is a temporary solution but at this rate, the BaseBenchmarkReports landing page will soon be filled with folders.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants