Compactor no longer compacts after ingester issues #9731
Unanswered
braamvh-nintex
asked this question in
Help and support
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi all. I have a situation where my compactors no longer compact properly. I get a lot of messages like these:
{"caller":"compactor.go:652","component":"compactor","err":"compaction: group 0@17241709254077376921-merge--1728475200000-1728482400000: gather index issues for block /data/compact/0@17241709254077376921-merge--1728475200000-1728482400000/01J9SBW4BEK40A5921XE632KWC: get all postings: decode postings: invalid size","level":"error","msg":"failed to compact user blocks","ts":"2024-10-23T12:09:56.936917013Z","user":"<removed>"}
The ingesters were incorrectly configured (without disks), and when I added that in, they started corrupting blocks. I finally managed to get a stable configuration for the ingesters, and it is possible they uploaded some corrupted data to the block store.
I went and deleted all the blocks (
01J9SBW4BEK40A5921XE632KWC
in the message above) and restarted the compactors, but it came up with a whole new list on the subsequent run. Am I on the right track, or is this not the correct approach? Is there a way to identify the corrupted blocks here? I am using an Azure storage container.Beta Was this translation helpful? Give feedback.
All reactions