Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvements for CI in GitHub Actions #1787

Merged
merged 189 commits into from
Dec 10, 2024
Merged

Conversation

jagthedrummer
Copy link
Contributor

@jagthedrummer jagthedrummer commented Nov 27, 2024

This is similar to the joint PR in the core repo`. Both PRs improve the CI workflow in their respective repos.

  • Adds the parallel_tests gem to split tests in CI instead of a custom script we were using previously.
  • Uses parallel_tests to not only split tests across "physical" nodes, but also parallelize test runs on each node. We default to 4 nodes, with 4 runners per node, for a total of 16 individual runners. In private repos we only do 2 runners per node (because private repos only get 2 CPU cores per node), for a total of 8 individual runners. Number of nodes and number of runners per node are configurable in the workflow.
  • Uses the runtime grouping option of parallel_tests to create test groups that should run in about the same amount of time. Previously we used random grouping and so sometimes one runner would end up with a bunch of slow tests and run times would be fairly uneven.
  • We use workflow artifacts to store runtime info about each test, and then in a later workflow we combine that data and store it in a cache, which is then used by subsequent test runs to optimize test grouping.
  • Uses bullet-train-co/parallel-test-dynamic-matrix action to dynamically construct the matrix of test runners based on the configuration (or defaults) mentioned above. This makes it much less fiddly and error prone to configure whatever parallelization you want.
  • Uses standardrb/standard-ruby-action instead of running standardrb manually. Using the official action makes it so that any linting failures are added as annotations to the workflow.
  • Instead of summarizing test status for each individual test node we now use workflow artifacts to store test status for each node and then we use a later workflow to combine the status data from all nodes into a single report.

Before we would show 4 different summaries:

CleanShot 2024-12-05 at 11 35 12

Now we show a single summary that includes all tests:

CleanShot 2024-12-05 at 11 22 37

When there are test failures they are collected and all reported together instead of being spread across the different summaries:

CleanShot 2024-12-05 at 11 21 18

  • Use simplecov to generate data about test coverage. We use workflow artifacts to save coverage data from each test node, and then a later workflow combines the coverage data and generates a summary. The summary includes expandable sections. One that shows coverage data for the top 10 least covered files, and another that show coverage by groups.

CleanShot 2024-12-10 at 11 38 06

The summary also includes a link to a downloadable coverage report. After downloading the report you can open it in your browser to get the full picture of the coverage situation.

CleanShot 2024-12-10 at 11 40 05

The 🚅 _ BT - Internal CI workflow got a similar update.

CleanShot 2024-12-10 at 11 53 26

@jagthedrummer jagthedrummer merged commit 907047b into main Dec 10, 2024
29 checks passed
@jagthedrummer jagthedrummer deleted the jeremy/parallel_tests branch December 10, 2024 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant