Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: Add caching / storing of bench report on default branch #290

Merged
merged 2 commits into from
Mar 17, 2022

Conversation

shahzadlone
Copy link
Member

@shahzadlone shahzadlone commented Mar 11, 2022

This PR starts uploading benchmark reports of every commit pushed (through PR) on the develop branch.

Next PR will:

  • use the more complicated artifact downloader (which enables us to find the artifacts of previous runs).
  • compare the current benchmark report with the latest artifact report stored on develop.

@shahzadlone shahzadlone self-assigned this Mar 11, 2022
@shahzadlone shahzadlone added the action/full-benchmark Runs the full / longer benchmark action. label Mar 11, 2022
@shahzadlone shahzadlone added this to the DefraDB v0.3 milestone Mar 11, 2022
@shahzadlone shahzadlone linked an issue Mar 11, 2022 that may be closed by this pull request
Copy link
Member

@jsimnz jsimnz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comments

Comment on lines 190 to 194
uses: actions/upload-artifact@v3
with:
name: bench-artifact
path: bench-artifact.txt
retention-days: 90
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this over-write the previous artifact? Since the "download" artifact uses the same name.

Seems like it would make sense to create two artifacts.

  • bench-artifact-develop.txt
  • bench-artifact-COMMIT_HASH.txt

Then we can keep their history and tie it to individual commits hashes

Copy link
Member Author

@shahzadlone shahzadlone Mar 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I was aiming to see if we can keep one file (the latest commit on develop branch's benchmark report) and always compare just with the report, which does mean we overwrite the latest on develop artifact (this could have issues if multiple runs are writing at the same time, but I just wanted to see it working barebones in this PR the rest was scoping for the next PR that does the actual comparisons).

Are you suggesting to use the same approach above but with bench-artifact-develop.txt and in addition store all previous commit reports that were pushed to develop as well with bench-artifact-COMMIT_HASH.txt (store previously stored artifact before overwriting) even if we don't use bench-artifact-COMMIT_HASH.txt in CI for anything? This can be nice quickly see bench mark report of any previous commit quickly.

Copy link
Contributor

@orpheuslummis orpheuslummis Mar 12, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two not-strong arguments in favor of keeping history:

  • non-zero/positive probability that benchmark information of current commits/PRs is useful in the future, e.g. to understand performance impact of some past changes to understand our "performance options" at a certain point
  • storage is cheap

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I was aiming to see if we can keep one file (the latest commit on develop branch's benchmark report) and always compare just with the report, which does mean we overwrite the latest on develop artifact (this could have issues if multiple runs are writing at the same time, but I just wanted to see it working barebones in this PR the rest was scoping for the next PR that does the actual comparisons).

Are you suggesting to use the same approach above but with bench-artifact-develop.txt and in addition store all previous commit reports that were pushed to develop as well with bench-artifact-COMMIT_HASH.txt (store previously stored artifact before overwriting) even if we don't use bench-artifact-COMMIT_HASH.txt in CI for anything? This can be nice quickly see bench mark report of any previous commit quickly.

Yupp, suggesting the additional per commit on develop benchmark artifact. Even if we're not using it yet it'll be useful soon, and as Orpheus said, storage is cheap

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup was intending to add that in the next PR because then there will be an artifact uploaded to cache already (when this PR gets merged).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might as well just add it to this PR, just a few extra lines.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this PR should just handle the uploading of the artifacts with their hashes on a develop push event. That is what the latest change on this PR is doing.

Next PR will have the fetching of a previous run's artifact logic and comparisons of the benchmark reports.

Copy link
Member

@jsimnz jsimnz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@shahzadlone shahzadlone merged commit dcc1978 into develop Mar 17, 2022
@shahzadlone shahzadlone deleted the lone/ci/use-artifacts-to-upload-benchmark branch March 17, 2022 03:01
shahzadlone added a commit to shahzadlone/defradb that referenced this pull request Feb 23, 2024
…twork#290)

This PR starts uploading benchmark reports of every commit pushed (through PR) on the develop branch.

Next PR will:
- use the artifact downloader (which enables us to find the artifacts of previous runs).
- compare the current benchmark report with the latest artifact report stored on develop.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
action/full-benchmark Runs the full / longer benchmark action.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cache full benchmark run on develop
3 participants