Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Writable Asset Cache #171

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Writable Asset Cache #171

wants to merge 1 commit into from

Conversation

ekcasey
Copy link
Member

@ekcasey ekcasey commented Jun 24, 2021

Signed-off-by: Emily Casey <ecasey@vmware.com>

The way caching currently works in CNB, the layer cache is scoped to a particular image and the cached layers are only restored during a rebuild of that exact image. There are good reasons for this. First of all, on the first build of a new app there is no way of knowing which layers might be helpful to restore. Because layers affect the behavior of subsequent buildpacks and may be directly included in the final app image (launch layers), the potential consequences of deliberate cache poisoning or accidental leakage of unexpected changes between images are high. Buildpacks general cannot predict the diffID of a given layer and therefore have no way to check the integrity of a restored layer. Therefore it is essential that layer caching remains scoped to rebuilds.

However, caching of assets, particularly if they are not restored to a layer, presents much less risk. Buildpacks generally known the hash of the asset they are attempting to download and can therefore check the integrity of a cached asset before using it in a layer.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
However, caching of assets, particularly if they are not restored to a layer, presents much less risk. Buildpacks generally known the hash of the asset they are attempting to download and can therefore check the integrity of a cached asset before using it in a layer.
However, caching of assets, particularly if they are not restored to a layer, presents much less risk. Buildpacks generally know the hash of the asset they are attempting to download and can therefore check the integrity of a cached asset before using it in a layer.


![flowchart](https://docs.google.com/drawings/d/1cY91fi9DGSAK9BKO8kcjQSXaNnE-9suk2gDfRgxeHlY/export/png)


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An example buildpack using this new $CNB_ASSET_CACHE would be very valuable in addition to this awesome flowchart.

@shresthaujjwal
Copy link

shresthaujjwal commented Jan 25, 2022

@elcasey I am very much interested in this RFC. I have a similar concern too. We are still on design phase to start implementation of buildpacks in our enterprise and we use Jenkins for our CiCd and I have the same concern. When we run our pack build there is no guarantee that the build will run on same Jenkins node as we have lot of cluster nodes and any one node can pick up the build process. Hopefully this RFC would solve our concerns as we need to be able to reuse the Docker cached layers across the clustered nodes.

Thanks

@buildpack-bot
Copy link
Member

Maintainers,

As you review this RFC please queue up issues to be created using the following commands:

/queue-issue <repo> "<title>" [labels]...
/unqueue-issue <uid>

Issues

(none)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants