Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cargo caching with forklift #2466

Merged
merged 38 commits into from
Dec 4, 2023
Merged

Cargo caching with forklift #2466

merged 38 commits into from
Dec 4, 2023

Conversation

AndWeHaveAPlan
Copy link
Contributor

This PR adds cargo caching feature with custom 'forklift' tool
Forklift acts as RUSTC_WRAPPER, intercepts rustc calls and stores produced artifacts in S3 bucket (see forklift readme for detailed description)

All settings are made in .forklift job's before_script and affect all jobs that extend .docker-env job

To disable feature set FORKLIFT_BYPASS variable to true in project settings in gitlab

@AndWeHaveAPlan AndWeHaveAPlan added R0-silent Changes should not be mentioned in any release notes A1-insubstantial Pull request requires no code review (e.g., a sub-repository hash update). labels Nov 23, 2023
@AndWeHaveAPlan AndWeHaveAPlan marked this pull request as ready for review November 23, 2023 14:13
@AndWeHaveAPlan AndWeHaveAPlan requested a review from a team as a code owner November 23, 2023 14:13
@rcny
Copy link
Contributor

rcny commented Nov 23, 2023

Well done! 🙏

@bkchr
Copy link
Member

bkchr commented Nov 29, 2023

Any numbers on the speed up for the test-linux-stable job?

@AndWeHaveAPlan
Copy link
Contributor Author

Any numbers on the speed up for the test-linux-stable job?

Unfortunately, not good enough. Build speedup is 5-25% (depending on the job). This is because the current version only caches crates from crates.io. There are still some issues with the substrate-wasm-builder cache

@bkchr
Copy link
Member

bkchr commented Dec 1, 2023

There are still some issues with the substrate-wasm-builder cache

And what is with path dependencies? That also doesn't work?

@AndWeHaveAPlan
Copy link
Contributor Author

And what is with path dependencies? That also doesn't work?

For simplier repos (such as cargo) everything seems to be ok. But for polkadot-sdk I get errors E0460 (found newer version of crate) and E0463 (can't find crate). I'm still investigating if it is only substrate-wasm-builder crates problem. Right now I'm working on the idea that forklift may mess up cached artifacts with "real" target dir and builder "subtargets" under some conditions

@AndWeHaveAPlan AndWeHaveAPlan merged commit e6b9da1 into master Dec 4, 2023
115 checks passed
@AndWeHaveAPlan AndWeHaveAPlan deleted the es/forklift-test branch December 4, 2023 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A1-insubstantial Pull request requires no code review (e.g., a sub-repository hash update). R0-silent Changes should not be mentioned in any release notes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants