Optimize modules release time #326
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We should run with `-m` option to enable parallel uploads. Additionally, we should run with either `-c` or `-i` instead of the default (which compares modification time). Since files are sync'd from a actions/checkout git clone, all of the files mtimes are set to the time of checkout, so the default behavior will be to copy every file every time.
rodaine
approved these changes
Nov 8, 2023
pkwarren
commented
Nov 8, 2023
| @@ -60,4 +60,4 @@ jobs: | |||
| - name: Setup gcloud | |||
| uses: google-github-actions/setup-gcloud@e30db14379863a8c79331b04a9969f4c1e225e0b | |||
| - name: Upload To Release Bucket | |||
| run: gsutil rsync -r modules/sync gs://buf-modules | |||
| run: gsutil -m rsync -c -r modules/sync gs://buf-modules | |||
There was a problem hiding this comment.
Global options (-m): https://cloud.google.com/storage/docs/gsutil/addlhelp/GlobalCommandLineOptions#options
Rsync options (-c): https://cloud.google.com/storage/docs/gsutil/commands/rsync#options
I really think it would be safe if we ran with -i which ignores mtime entirely and just skips any files which exist at the destination already, however we can try starting with -c to see what the impact on release time is. Right now I've confirmed that every time it copies all of the files.
There was a problem hiding this comment.
Actually -i won't work for files like state.json that change with each update. It would only work for the CAS files.
|
Lovely! thanks so much |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We should run with
-moption to enable parallel uploads. Additionally, we should run with either-cor-iinstead of the default (which compares modification time). Since files are sync'd from a actions/checkout git clone, all of the files mtimes are set to the time of checkout, so the default behavior will be to copy every file every time.