Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Precompilation cache invalidation mtime heuristic always rounds down #42842

Open
twavv opened this issue Oct 29, 2021 · 3 comments
Open

Precompilation cache invalidation mtime heuristic always rounds down #42842

twavv opened this issue Oct 29, 2021 · 3 comments

Comments

@twavv
Copy link
Contributor

twavv commented Oct 29, 2021

This is related to #13606.

Ultimately, it seems to be based on how Golang's archive/tar library handles rounding mtimes. I think(?) it the past it simply truncated them, but now it rounds to the nearest second. (Ultimately this restriction comes from trying to encode the tar file in the ustar format which is more space efficient but also worse and only encodes mtimes in seconds).

julia/base/loading.jl

Lines 1916 to 1922 in c054dbc

# Issue #13606: compensate for Docker images rounding mtimes
# Issue #20837: compensate for GlusterFS truncating mtimes to microseconds
ftime = mtime(f)
if ftime != ftime_req && ftime != floor(ftime_req) && ftime != trunc(ftime_req, digits=6)
@debug "Rejecting stale cache file $cachefile (mtime $ftime_req) because file $f (mtime $ftime) has changed"
return true
end

A simple solution would just be to change the latter chunk (trunc, presumably there for the GlusterFS issue) to ftime != ceil(ftime_req). I can open that PR if it sounds good.

Admittedly, I ran into this issue working well off the beaten path (doing very wild things with containers), and I'm not sure why this doesn't happen with regular Docker images, but shrug, I don't think this would be a big deal to fix.

@twavv twavv changed the title Precompile mtime heuristic is slightly wrong Precompilation cache invalidation mtime heuristic always rounds down Oct 29, 2021
@omus
Copy link
Member

omus commented Nov 25, 2022

Seems to me that we probably shouldn't be invalidating the cache files based upon the mtime alone but rather use the detection of an mtime change to initiate a checksum. For systems that modify the mtime in some way this should avoid false invalidations.

@vtjnash
Copy link
Member

vtjnash commented Nov 25, 2022

It is more that we do not want to do a lot of file reads, but it is also shocking how many projects cannot manage to store a simple number accurately.

@DilumAluthge
Copy link
Member

Seems to me that we probably shouldn't be invalidating the cache files based upon the mtime alone but rather use the detection of an mtime change to initiate a checksum. For systems that modify the mtime in some way this should avoid false invalidations.

Cross-ref #45541

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants