Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvements for CI in GitHub Actions #979

Merged
merged 70 commits into from
Dec 10, 2024
Merged

Conversation

jagthedrummer
Copy link
Contributor

@jagthedrummer jagthedrummer commented Dec 5, 2024

This is very similar to the joint PR in the starter repo. Both PRs improve the CI workflow in their respective repos.

  • Uses parallel_tests (installed in the starter repo) to not only split tests (starter repo minitest) across "physical" nodes, but also parallelize test runs on each node. We're using 4 nodes, with 4 runners per node, for a total of 16 individual runners.
  • Uses the runtime grouping option of parallel_tests to create test groups that should run in about the same amount of time. Previously we used random grouping and so sometimes one runner would end up with a bunch of slow tests and run times would be fairly uneven.
  • We use workflow artifacts to store runtime info about each test, and then in a later workflow we combine that data and store it in a cache, which is then used by subsequent test runs to optimize test grouping.
  • Uses bullet-train-co/parallel-test-dynamic-matrix action to dynamically construct the matrix of test runners based on the configuration (or defaults) mentioned above. This makes it much less fiddly and error prone to configure whatever parallelization you want.
  • Use a custom job to calculate the "gem matrix" for running individual gem tests. Previously we had hard-coded the list of gems into the workflow. The dynamic matrix means that when new gems are added in the future they'll be automatically picked up by the CI workflow.
  • Uses standardrb/standard-ruby-action instead of running standardrb manually. Using the official action makes it so that any linting failures are added as annotations to the workflow.
  • Instead of summarizing test status for each individual test node we now use workflow artifacts to store test status for each node and then we use a later workflow to combine the status data from all nodes into a single report.

Before we would show dozens of different summaries:

CleanShot 2024-12-10 at 11 24 51

Now we show a single summary that includes all tests (starter repo minitest, starter repo super scaffolding test, and individual test inside each gem):

CleanShot 2024-12-10 at 11 18 11

When there are test failures they are collected and all reported together instead of being spread across the different summaries:

CleanShot 2024-12-10 at 11 34 26

  • Use simplecov to generate data about test coverage. We use workflow artifacts to save coverage data from each test node, and then a later workflow combines the coverage data and generates a summary.

CleanShot 2024-12-10 at 11 22 35

The summary includes expandable sections. One that shows coverage data for the top 10 least covered files, and another that show coverage by groups. In the core repo the grouping is for the individual gems and the starter repo.

CleanShot 2024-12-10 at 11 19 24

The summary also includes a link to a downloadable coverage report. After downloading the report you can open it in your browser to get the full picture of the coverage situation.

CleanShot 2024-12-10 at 11 56 31

@jagthedrummer jagthedrummer changed the title Use parallel_tests in CI Improvements for CI in GitHub Actions Dec 10, 2024
@jagthedrummer jagthedrummer merged commit fabd8d1 into main Dec 10, 2024
34 checks passed
@jagthedrummer jagthedrummer deleted the jeremy/parallel_tests branch December 10, 2024 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant