-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
buildsys: add CI/CD hack, fix mtime for external files #345
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cbgbt
approved these changes
Aug 20, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚀
mgsharm
approved these changes
Aug 23, 2024
For local builds, the detection logic around when to rebuild crates works fairly well outside of edge cases like upgrading Rust. However, the same is not true for reusing cached builds with CI/CD. `cargo` relies on file modification times, and Git does not preserve these across checkouts, making it difficult to get unmodified files back to the state that `cargo` expects in order to avoid a rebuild. In addition, the fingerprint that `cargo` assigns for a given crate is not consistent across checkouts or changes to the Rust toolchain, which again leads to unnecessary rebuilds. To sidestep these challenges, add a `BUILDSYS_CICD_HACK` environment variable that causes `buildsys` to exit before launching any Docker builds, and after downloading external files and emitting the change detection commands. This assumes that the build output directory is complete and current, which is not possible to prove without letting the build run. If that assumption holds, then the result is that the crates produced by `cargo` will be regenerated and will be primed to track subsequent changes. Signed-off-by: Ben Cressey <bcressey@amazon.com>
External files may end up listed in a package's spec in some cases, such as patches or the generated Go module bundles. These files are created at arbitrary times relative to the start of the build, since they may take a long time to download or prepare. This can cause spurious rebuilds since `cargo` assumes that any file tracked for changes will not have been modified after the file that stores output for the build. To avoid these rebuilds, set the modification time for external files and generated bundles to match the `Cargo.toml` manifest. Signed-off-by: Ben Cressey <bcressey@amazon.com>
First force push handles the rebase. Second force push adds some documentation, and adjusts the argument parsing to take a bool instead of an string value that's ignored.
Testing:
|
cbgbt
approved these changes
Aug 30, 2024
sam-berning
approved these changes
Aug 30, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue number:
N/A
Description of changes:
Two
buildsys
changes, one potentially controversial, one hopefully not.The first is the
BUILDSYS_CICD_HACK
variable - naming suggestions wanted! - to allowcargo
to recreate crates without actually running expensive package builds. It's a hack because it assumes thebuild
directory is fully populated; if it's not, then bad things will happen the next time a package is rebuilt.The second fixes an edge case related to external files or bundled Go modules; these can end up being tracked by
cargo
for changes, so we need to ensure that their modification time is no later than the start of the build.Cargo.toml
is already tracked for changes and is a useful sentinel for this purpose.Testing done:
Tested in bottlerocket-os/bottlerocket-core-kit#67. With both changes, I was able to reuse cached package builds, touch the files modified by a pull request, and observe that only the affected packages were rebuilt.
Terms of contribution:
By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.