Fix rust caching and duplicate buildpack compilation #91
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is a restructuring of the whole release process to eliminate the issues outlines in #48 (buildpacks compiled multiple times could result in non-determinism) and #49 (rust caching isn't working due to non-unique keys). Many of the overall steps are still the same but the `detect` job has been replaced by a `compile` job which front loads all the rust and buildpack compilation and cached the compiled buildpack folder for later use. Compilation was previously handled in the `package` job which is no longer needed. To support this new flow, a `buildpack_output_dir` value had to be added to the generated buildpack matrix. This also allowed some simplification in terms of the assets cached since the OCI image and CNB file can be generated directly from the compiled `buildpack_output_dir` we don't need to store those values in the cache anymore.
edmorley
reviewed
Jul 27, 2023
There was a problem hiding this comment.
Thank you for doing this! I love the new combined shared workflow idea too - as you say, it should reduce the boilerplate in each buildpack's repo :-)
I presume the rollout plan for this is to merge this PR, switch the repos over to using the new workflow, then remove the old separated workflows afterwards? (If so, that sounds good to me - I was just initially wondering why the old workflows weren't deleted in this PR.)
|
@edmorley re: rollout plan - you are correct. merge this PR, switch the repos over to using the new workflow, then remove the old separated workflows afterwards. |
Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com>
Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com>
Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com>
edmorley
reviewed
Jul 28, 2023
edmorley
approved these changes
Jul 28, 2023
colincasey
added a commit
to heroku/buildpacks-nodejs
that referenced
this pull request
Aug 1, 2023
Incorporates changes from heroku/languages-github-actions#91
colincasey
added a commit
to heroku/buildpacks-jvm
that referenced
this pull request
Aug 1, 2023
Incorporates changes from heroku/languages-github-actions#91
colincasey
added a commit
to heroku/buildpacks-ruby
that referenced
this pull request
Aug 1, 2023
Incorporates changes from heroku/languages-github-actions#91
colincasey
added a commit
to heroku/buildpacks-go
that referenced
this pull request
Aug 1, 2023
Incorporates changes from heroku/languages-github-actions#91
colincasey
added a commit
to heroku/buildpacks-php
that referenced
this pull request
Aug 1, 2023
Incorporates changes from heroku/languages-github-actions#91
edmorley
pushed a commit
to heroku/buildpacks-nodejs
that referenced
this pull request
Aug 2, 2023
Switches to the new combined workflow approach added in: heroku/languages-github-actions#91
edmorley
pushed a commit
to heroku/buildpacks-jvm
that referenced
this pull request
Aug 2, 2023
Switches to the new combined workflow approach added in: heroku/languages-github-actions#91
edmorley
pushed a commit
to heroku/buildpacks-ruby
that referenced
this pull request
Aug 2, 2023
Switches to the new combined workflow approach added in: heroku/languages-github-actions#91
edmorley
pushed a commit
to heroku/buildpacks-go
that referenced
this pull request
Aug 2, 2023
Switches to the new combined workflow approach added in: heroku/languages-github-actions#91
edmorley
pushed a commit
to heroku/buildpacks-php
that referenced
this pull request
Aug 2, 2023
Switches to the new combined workflow approach added in: heroku/languages-github-actions#91
colincasey
added a commit
that referenced
this pull request
Aug 3, 2023
We can remove these separated workflows now that the following PRs have been merged: - heroku/buildpacks-jvm#554 - heroku/buildpacks-php#14 - heroku/buildpacks-go#123 - heroku/buildpacks-ruby#183 - heroku/buildpacks-nodejs#609 - heroku/buildpacks-python#58 - heroku/buildpacks-procfile#156
colincasey
added a commit
that referenced
this pull request
Aug 4, 2023
* Post PR #91 Cleanup We can remove these separated workflows now that the following PRs have been merged: - heroku/buildpacks-jvm#554 - heroku/buildpacks-php#14 - heroku/buildpacks-go#123 - heroku/buildpacks-ruby#183 - heroku/buildpacks-nodejs#609 - heroku/buildpacks-python#58 - heroku/buildpacks-procfile#156 * PR feedback
colincasey
added a commit
that referenced
this pull request
Aug 4, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a restructuring of the whole release process to eliminate the issues outlines in #48 (buildpacks compiled multiple times could result in non-determinism) and #49 (rust caching isn't working due to non-unique keys).
Many of the overall steps are still the same but the
detectjob has been replaced by acompilejob which front loads all the rust and buildpack compilation and cached the compiled buildpack folder for later use. Compilation was previously handled in thepackagejob which is no longer needed.To support this new flow, a
buildpack_output_dirvalue had to be added to the generated buildpack matrix. This also allowed some simplification in terms of the assets cached since the OCI image and CNB file can be generated directly from the compiledbuildpack_output_dirwe don't need to store those values in the cache anymore.Fixes #48 (W-13738777), fixes #49 (W-13738739)
Note - I discovered I was mistaken about how strategies and shared workflows interact. Turns out you can use them. The GitHub Actions docs make it sound like you can't and I must have messed up when I initially tried this. Using this new shared workflow should simplify the calling
release.ymlfiles in our buildpack repos 😄We'll also need to be careful with changes to buildpack output locations like heroku/libcnb.rs#583 and ensure these still line up with expectations in our release process.