Investigate FIFO asset cache cleaner with Pack creation time behavior #302
Comments
|
Comment is a work in progress. I'm trying to de-tangle all the information about build reproducibility mtime Application use of mtimeLibraries and applications use the CNB project mtime implicationsThe cloud native buildpack project generates OCI images. One desired property of an OCI image is deterministic builds. That is, given the same inputs, the project produces the same image byte-for-byte. Timestamps complicate determinisim. Another important feature of CNBs is rebasing https://buildpacks.io/docs/for-platform-operators/how-to/integrate-ci/pack/cli/pack_rebase/. Rebasing allows for updating system dependencies (like a version of curl, for example) without the need to completely re-build the entire image. CNB Features
[...]
History
|
Reproducing cache mtime behaviorShort: I was wrong, mtime of files from the cache is preserved, it's only runtime that's zero-d Long: Make a buildpackUsing a buildpack that prints the mtime of the app dir and caches it. Here's the buildpack: Make an "app" with some files with an mtimeMake some files, note that they have different mtimes, about 1 minute apart: This was run on a mac First build of the buildpackWe can see that mtime of the incoming files is correct. They were modifed about 1 minute apart from each other: We can see the cache dir afterwards uses the correct timestamps, they were modified one minute apart: Build with cacheWe see the mtime is preserved |
|
Therefore: I think this is maybe not an issue. The mtime of the files in the cache seem to be preserved. The launch mtime will be zero-d but that doesn't affect us. I was only worried about cache cleaning. One caveat is that we want to make sure any files copied have a preserved mtime i.e. I need to validate that we're copying and preserving mtime in the buildpack |
|
fs_extras does not explicitly copy mtime there is a PR implementing this behavior webdesus/fs_extra#53. The library One path forward is to copy files and then in a second operation, copy the mtimes |
Noted in heroku/buildpacks#22,
packmucks with creation time.The Ruby CNB supports caching assets from
rake assets:precompile, but we need a way to expire them. Beyond asset performance, it's also used for features such as sprocket's "last 3 versions" of an asset being made available. (In the asset pipeline when you modify a file, it generates a new output with a new SHA in the filename. But things like emails might reference older asset SHAs so you want to keep the old ones around for a little while, this is accomplished by storing assets generated in the cache between builds.)Likewise, we need some way to invalidate cache entries or the cache size would grow indefinitely. The classic buildpack caps the memory limit of the asset directory and when it hits that limit, it deletes the oldest files first as newer files are more likely to still be used or referenced.
The problem (as I'm guessing) is that if
packis messing with creation time then it's effectively giving all files a very similar creation time even if they were created months/years apart. This could mean that the most recently generated assets are now cleared from cache instead of the oldest ones which will result in random asset lookup failures at runtime that will be difficult to diagnose and debug.TODO
The text was updated successfully, but these errors were encountered: