Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalidate the cache when the stack changes #21

Merged
merged 1 commit into from
Jun 21, 2024

Conversation

edmorley
Copy link
Member

@edmorley edmorley commented Jun 19, 2024

Previously the buildpack did not invalidate the cache when the stack changed, which causes the errors seen in #19. This will be particularly an issue when Heroku-24 is GAed and Heroku-20 deprecated, since there will be a number of apps changing stack for the first time after switching to this buildpack.

Now, the stack version is stored in the cache and if it differs from the current stack version (or the version file is missing since the cache is from an older version of this buildpack), the cache will be purged.

This implementation is an improved version of what's implemented for the generic APT buildpack (I'll be backporting those improvements over to that buildpack shortly):
https://github.com/heroku/heroku-buildpack-apt/blob/4eb4b35d35d0178e5cef73d6998a26ba2c9bbf17/bin/compile#L42-L86

Longer term we should probably re-evaluate whether we need the cache at all, since from my testing locally it offered very little benefit, and if anything causes a number of issues (particularly when several APT-using buildpacks are set on an app at the same time, given the shared Debian archives directory and install strategy).

Fixes #19.
GUS-W-16045454.

@edmorley edmorley self-assigned this Jun 19, 2024
@edmorley
Copy link
Member Author

I've tested all four scenarios:

Clean cache build using this branch

$ mkdir chrome-test && cd $_
$ git init && touch foo && git add -A && git commit -m '.'
$ h create --buildpack 'https://github.com/heroku/heroku-buildpack-chrome-for-testing#edmorley/cache-invalidation-stack'
$ git push heroku main
...
remote: -----> Installing Chrome dependencies
remote:        Updating APT package index
...

Warm cache build showing cache reuse

$ mkdir chrome-test && cd $_
$ git init && touch foo && git add -A && git commit -m '.'
$ h create --buildpack 'https://github.com/heroku/heroku-buildpack-chrome-for-testing#edmorley/cache-invalidation-stack'
$ git push heroku main
$ echo >> foo && git commit -am '.'
$ git push heroku main
...
remote: -----> Installing Chrome dependencies
remote:        Reusing APT cache
remote:        Updating APT package index
...

Warm cache build showing cache invalidation on stack change

$ mkdir chrome-test && cd $_
$ git init && touch foo && git add -A && git commit -m '.'
$ h create --stack heroku-22 --buildpack 'https://github.com/heroku/heroku-buildpack-chrome-for-testing#edmorley/cache-invalidation-stack'
$ git push heroku main
$ h stack:set heroku-20
$ echo >> foo && git commit -am '.'
$ git push heroku main
...
remote: -----> Installing Chrome dependencies
remote:        Clearing APT cache since stack changed from heroku-22 to heroku-20
remote:        Updating APT package index
...

Migration of existing cache written by older version of this buildpack

$ mkdir chrome-test && cd $_
$ git init && touch foo && git add -A && git commit -m '.'
$ h create --buildpack https://github.com/heroku/heroku-buildpack-chrome-for-testing
$ git push heroku main
$ h buildpacks:set 'https://github.com/heroku/heroku-buildpack-chrome-for-testing#edmorley/cache-invalidation-stack'
$ echo >> foo && git commit -am '.'
$ git push heroku main
...
remote: -----> Installing Chrome dependencies
remote:        Clearing APT cache since it's missing the stack version metadata file.
remote:        Updating APT package index
...

@edmorley edmorley marked this pull request as ready for review June 19, 2024 13:45
Copy link
Member

@mars mars left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonderful, this will avoid so many issues as customers migrate to newer stacks!

Previously the buildpack did not invalidate the cache when the stack
changed, which causes the errors seen in #19. This will be particularly
an issue when Heroku-24 is GAed and Heroku-20 deprecated, since
there will be a number of apps changing stack for the first time after
switching to this buildpack.

Now, the stack version is stored in the cache and if it differs from the
current stack version (or the version file is missing since the cache is
from an older version of this buildpack), the cache will be purged.

This implementation is an improved version of what's implemented
for the generic APT buildpack (I'll be backporting those improvements
over to that buildpack shortly):
https://github.com/heroku/heroku-buildpack-apt/blob/4eb4b35d35d0178e5cef73d6998a26ba2c9bbf17/bin/compile#L42-L86

Longer term we should probably re-evaluate whether we need the cache
at all, since from my testing locally it offered very little benefit, and if
anything causes a number of issues (particularly when several APT-using
buildpacks are set on an app at the same time, given the shared Debian
archives directory and install strategy).

Fixes #19.
@edmorley edmorley force-pushed the edmorley/cache-invalidation-stack branch from 52f05de to 2736ead Compare June 21, 2024 10:11
@edmorley edmorley merged commit 0d6dabc into main Jun 21, 2024
3 checks passed
@edmorley edmorley deleted the edmorley/cache-invalidation-stack branch June 21, 2024 10:13
@edmorley
Copy link
Member Author

Released in:
https://github.com/heroku/heroku-buildpack-chrome-for-testing/releases/tag/1.0.7

$ heroku buildpacks:publish heroku-community/chrome-for-testing 1.0.7
Publishing buildpack heroku-community/chrome-for-testing... done
Publishing buildpack was successful

edmorley added a commit to heroku/heroku-buildpack-apt that referenced this pull request Jun 24, 2024
Now:
- If the stack version file is missing, we don't assume the cached files
  belong to the current stack (since that could cause breakage), and
  instead invalidate the cache. (The cache actually offers little benefit
  in practice for this buildpack, so invalidating is cheap.)
- The whole `${CACHE_DIR}/apt` directory is removed rather than
  only the `${CACHE_DIR}/apt/cache` directory. This ensures the APT
  indexes and other files from old stack versions are cleaned up too.
- Some build output logs have been adjusted to be more accurate.
- A test has been added for cache re-use, since it wasn't previously
  tested.

These papercuts were noticed whilst working on:
heroku/heroku-buildpack-chrome-for-testing#21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cache isn't invalidated when the stack changes
2 participants