-
Notifications
You must be signed in to change notification settings - Fork 377
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make running benchmarks in CI more consistent #526
Closed
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pawelchcki
force-pushed
the
ci/backport_benchmark_changes
branch
from
August 24, 2018 14:43
3e53818
to
4ebf8d4
Compare
pawelchcki
added
core
Involves Datadog core libraries
dev/tooling
Involves tools (e.g. Rubocop, CodeCov)
labels
Aug 24, 2018
This was referenced Sep 4, 2018
pawelchcki
force-pushed
the
ci/backport_benchmark_changes
branch
7 times, most recently
from
September 12, 2018 12:26
4b08a02
to
23c718d
Compare
pawelchcki
force-pushed
the
ci/backport_benchmark_changes
branch
from
October 1, 2018 22:46
bad39a3
to
bb96c7b
Compare
Just a minor request but could we squash some of these commits? There are a lot of them, haha. |
pawelchcki
force-pushed
the
ci/backport_benchmark_changes
branch
from
October 2, 2018 15:35
befcbb9
to
df83e69
Compare
pawelchcki
force-pushed
the
ci/backport_benchmark_changes
branch
3 times, most recently
from
October 2, 2018 16:38
e646e86
to
6aa0521
Compare
Rename Job_default to default_workflow
pawelchcki
force-pushed
the
ci/backport_benchmark_changes
branch
from
October 2, 2018 16:44
6aa0521
to
cc6332a
Compare
pawelchcki
force-pushed
the
ci/backport_benchmark_changes
branch
from
October 2, 2018 16:49
cc6332a
to
24a9ded
Compare
pawelchcki
force-pushed
the
ci/backport_benchmark_changes
branch
3 times, most recently
from
October 3, 2018 14:35
0820c3b
to
e817ee7
Compare
Can you take another look at this PR @delner? |
This is a very old PR and out of date. Will close for now. |
delner
added
dev/ci
Involves CircleCI, GitHub Actions, or GitLab
wont-fix
A pull request or issue that will not be addressed with changes
and removed
ci-cache-dependencies
labels
Apr 21, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
core
Involves Datadog core libraries
dev/ci
Involves CircleCI, GitHub Actions, or GitLab
dev/tooling
Involves tools (e.g. Rubocop, CodeCov)
wont-fix
A pull request or issue that will not be addressed with changes
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR was made in an effort to isolate and measure performance improvements.
Background:
Due to complex nature our current benchmark have high variability in results. This means that its hard/impossible to compare the results looking for improvements or regressions.
Description:
This PR allows running multiple benchmarks in parallel and collecting + processing the results.
Basically using fork + join pattern.
Additional improvements:
In addition this PR include a few improvements to CI like:
Thus in addition to benchmark this PR optimizes the CI runtime ~2x.
Benchmarks:
All benchmarks are optional not to use too much resources, additionally they are currently limited in how much resources they use. I.e. they are split into 5 parts each with parallelism of 1. If we've had more resources we might increase parallelism to run more jobs and make our data even more trustworthy.