fix store: handle invalid cache block dir #1505
Conversation
|
CI failing in master, will rebase once fixed |
FUSAKLA
force-pushed
the
fus-store-handle-invalid-block
branch
from
cff24a3
to
0b60b5f
Compare
There was a problem hiding this comment.
Hi there, thanks for the contribution! The code itself looks okay and I get the idea but I have taken a look at your original ticket. As I understand it, all of the data looks okay in the remote object storage but you have empty directories in your local disk. However, how could this be if we run SyncBlocks() before? It seems like the correct place to fix here is this: https://github.com/thanos-io/thanos/blob/master/pkg/store/bucket.go#L1191 - we should probably add this check there. @FUSAKLA WDYT?
There was a problem hiding this comment.
Sorry for lag @FUSAKLA
I think I fully agree with @GiedriusS
It looks like our loadMeta is just broken as it assumes that if dir is present then meta.json is present as well. We should rather check for meta.json directly.
This thing is happening because if you restart store in wrong moment (when it syncs blocks meta.json), the directory can be created by meta.json not downloaded. Thanks for fixing this bug - it's quite edge case bug, but serious as we skip data!
Thanks!
|
Hi, yes @GiedriusS that totally makes sense. Thanks for the suggestion will refactor right away. Just to add to the rarity. It's happening quite often in our production since upgrading to the 0.7.0 which is weird, but this should solve it. |
Signed-off-by: Martin Chodur <m.chodur@seznam.cz>
FUSAKLA
force-pushed
the
fus-store-handle-invalid-block
branch
from
0b60b5f
to
c58038a
Compare
|
@GiedriusS Should be moved as you suggested, PTAL if that is what you meant. |
pkg/block/block.go
Outdated
| @@ -153,3 +153,10 @@ func IsBlockDir(path string) (id ulid.ULID, ok bool) { | |||
| id, err := ulid.Parse(filepath.Base(path)) | |||
| return id, err == nil | |||
| } | |||
|
|
|||
| func HasMetaFile(blockPath string) bool { | |||
There was a problem hiding this comment.
I am against this shallow function - such helper is not needed - the oneliner below is exactly enough for this.
In the same way, there is no Max(a, b) function is go standard library. You just do this one if on your own (:
What do you think, can we just inline this logic?
There was a problem hiding this comment.
We could do what @bwplotka suggests, it might be neater. 👍
There was a problem hiding this comment.
I eventually simplified it to checking directly for the meta file which should also provide the check for the directory existence right away. The creation of dir is agnostic to it's existence and new downloaded files should overwrite those existing in the block dir.
Thanks @bwplotka for pointing this out I just saw the IsBlockDir func so I somehow followed the pattern.
PTAL if this is ok with you.
Signed-off-by: Martin Chodur <m.chodur@seznam.cz>
* fix store: handle invalid cache block dir Signed-off-by: Martin Chodur <m.chodur@seznam.cz> * CR: simplify empty store cache block validation logic Signed-off-by: Martin Chodur <m.chodur@seznam.cz>
fixes #1504
Changes
meta.jsonon startup