Refactor CI workflow - bring back reasonable CI runtime #6461
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
snazy
added
pr-docker
|
Sample workflow runs:
|
snazy
force-pushed
the
unified-workflow-gradle-action
branch
from
276478e
to
83c19dd
Compare
dimas-b
previously approved these changes
Apr 3, 2023
| matrix: | ||
| python-version: ['3.7', '3.8', '3.9', '3.10'] | ||
| python-version: ['3.8', '3.9', '3.10', '3.7'] |
There was a problem hiding this comment.
3.8 starts first (b/c it has "all the tox checks") - in case there are too many GH WF jobs running, you get that result before 3.9, 3.10, 3.7
.github/workflows/ci.yml
Outdated
|
|
||
| # Unifies main + PR workflow. | ||
| # | ||
| # The unified CI workflow consists of 4 "stages": |
There was a problem hiding this comment.
nit: there are only 3 list items below 🤷
Unifies most main + PR workflows into a single workflow. The unified CI workflow consists of 2 "stages": * Checks - test, intTest, NesQuEIT, etc * Finalize - a "success" dummy job + a "save to github-cache" job Utilizes the Gradle build cache for all stages. The updated build cache of the jobs in the checks stage are saved as artifacts (with the minimum retention period). The updated build cache is pushed back to GigHub's cache when the checks have successfully finished. Java CI runs against Java 11 and Java 17, where it is meaningful. (Spark + Deltalake tests always run against 11, so not run against Java 17.) Some checks also run against the latest Java version. Codecov was not added to the new workflow, it wasn't working for quite a while now or produced wrong results. Build logs and reports are not archived. Test results and relevant logs are available via Gradle build scans. Windows + macOS workflows are not included in the unified workflow. There is also another [Gradle cache action](https://github.com/burrunan/gradle-cache-action), which utilizes the GitHub's cache like a remote Gradle cache. However, that puts too much load (requests) against GitHub's cache, which in turn throttles our CI and responds with HTTP/429 (Too many requests). See [this issue](burrunan/gradle-cache-action#66). Fixes projectnessie#6365
snazy
force-pushed
the
unified-workflow-gradle-action
branch
from
83c19dd
to
47d2aca
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Unifies most main + PR workflows into a single workflow.
The unified CI workflow consists of 2 "stages":
Checks are split into multiple jobs, which brings back CI workflow runtimes of < 20 minutes for small-ish changes, up to 35 minutes at worst (assuming that all jobs are not delayed by GH concurrent job limits).
Utilizes the Gradle build cache as much as possible. The updated build cache of the jobs in the checks stage are saved as artifacts (with the minimum retention period). The updated build cache is pushed back to GigHub's cache when the checks have successfully finished.
Java CI runs against Java 11. The workflow can be easily adopted to add Java 17 for some (or all) jobs, but that would effectively double the amount of concurrent jobs, and there is a hard limit imposed by GitHub.
Spark + Deltalake tests always run against 11 as configured in the build scripts.
Codecov was not added to the new workflow, it wasn't working for quite a while now or produced wrong results.
Build logs and reports are not archived. Test results and relevant logs are available via Gradle build scans.
Windows + macOS workflows are not included in the unified workflow.
There is also another Gradle cache
action, which utilizes the GitHub's cache like a remote Gradle cache. However, that puts too much load (requests) against GitHub's cache, which in turn throttles our CI and responds with HTTP/429 (Too many requests). See this issue.
Fixes #6365