chore: ci reorganize dag #2529
Merged
chore: ci reorganize dag #2529
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: universalmind303 <cory.grinstead@gmail.com>
|
The last build in this branch exhibits some more worst-case behavior: the main build takes closer to 5 minutes (cache is less relevant because other branches are building concurrently.) Seeing these results makes me inclined to do per-branch caches, but need to experiment in practice. |
scsmithr
approved these changes
Jan 30, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Right we essentially compile four or five times (build+2 bindings+unit-tests+clippy). and they can't usefully share artifacts, but we insert a dependency between the build step and everything else so that if compile breaks we don't run anything else. A lot of things (including, I believe, the unit-tests) depend on the glaredb binary, so that makes sense but the bindings can build in parallel with the main cli binary.
The challenge/impediment, now, is the unit-test->glaredb dependency (integration tests in the glaredb crate). If that can be built and run separately, we can run it with the pytests or a new task and wall clock times would be in the 5-7 minute range for the entire pipeline.
The big downside is that if you push a broken compile that doesn't also break the formater (so many typos and syntax issues will still be caught,) then it has to fail four times, but this is classically a good trade in terms of increasing developer responsiveness. The other downside, is the first build after a Cargo.lock change (so no cache hits) will take more build time.