Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make running benchmarks in CI more consistent #526

Closed
wants to merge 11 commits into from

Conversation

pawelchcki
Copy link
Contributor

This PR was made in an effort to isolate and measure performance improvements.

Background:

Due to complex nature our current benchmark have high variability in results. This means that its hard/impossible to compare the results looking for improvements or regressions.

Description:

This PR allows running multiple benchmarks in parallel and collecting + processing the results.

Basically using fork + join pattern.

Additional improvements:

In addition this PR include a few improvements to CI like:

  • postgres using RAM disk as DB store
  • optimized redis
  • optimized mongodb
  • cached bundler install and apprisal install - this shaves off 1-4 minutes of off each build
  • removed checkout build - instead every build does its own git checkout + workspace is used to transfer data downstream in builds

Thus in addition to benchmark this PR optimizes the CI runtime ~2x.

Benchmarks:

All benchmarks are optional not to use too much resources, additionally they are currently limited in how much resources they use. I.e. they are split into 5 parts each with parallelism of 1. If we've had more resources we might increase parallelism to run more jobs and make our data even more trustworthy.

@pawelchcki pawelchcki self-assigned this Aug 24, 2018
@pawelchcki pawelchcki force-pushed the ci/backport_benchmark_changes branch from 3e53818 to 4ebf8d4 Compare August 24, 2018 14:43
@pawelchcki pawelchcki added core Involves Datadog core libraries dev/tooling Involves tools (e.g. Rubocop, CodeCov) labels Aug 24, 2018
@pawelchcki pawelchcki requested a review from delner August 24, 2018 16:20
@pawelchcki pawelchcki force-pushed the ci/backport_benchmark_changes branch 7 times, most recently from 4b08a02 to 23c718d Compare September 12, 2018 12:26
@pawelchcki pawelchcki force-pushed the ci/backport_benchmark_changes branch from bad39a3 to bb96c7b Compare October 1, 2018 22:46
@delner
Copy link
Contributor

delner commented Oct 2, 2018

Just a minor request but could we squash some of these commits? There are a lot of them, haha.

@pawelchcki pawelchcki force-pushed the ci/backport_benchmark_changes branch from befcbb9 to df83e69 Compare October 2, 2018 15:35
@pawelchcki pawelchcki force-pushed the ci/backport_benchmark_changes branch 3 times, most recently from e646e86 to 6aa0521 Compare October 2, 2018 16:38
@pawelchcki pawelchcki force-pushed the ci/backport_benchmark_changes branch from 6aa0521 to cc6332a Compare October 2, 2018 16:44
@pawelchcki pawelchcki force-pushed the ci/backport_benchmark_changes branch from cc6332a to 24a9ded Compare October 2, 2018 16:49
@pawelchcki pawelchcki force-pushed the ci/backport_benchmark_changes branch 3 times, most recently from 0820c3b to e817ee7 Compare October 3, 2018 14:35
@pawelchcki
Copy link
Contributor Author

Can you take another look at this PR @delner?

@delner
Copy link
Contributor

delner commented Apr 16, 2021

This is a very old PR and out of date. Will close for now.

@delner delner closed this Apr 16, 2021
@delner delner added dev/ci Involves CircleCI, GitHub Actions, or GitLab wont-fix A pull request or issue that will not be addressed with changes and removed ci-cache-dependencies labels Apr 21, 2021
@GustavoCaso GustavoCaso deleted the ci/backport_benchmark_changes branch October 11, 2023 11:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Involves Datadog core libraries dev/ci Involves CircleCI, GitHub Actions, or GitLab dev/tooling Involves tools (e.g. Rubocop, CodeCov) wont-fix A pull request or issue that will not be addressed with changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants