-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[bloom-compactor] Fix: blooms returning same checksum with different fp/ts params. #11720
Conversation
I feel like this misses the mark a bit. Seeing the same checksums after summing the content (blooms) and migrating to checksumming the metadata (index) just seems like we had a bug in the previous checksum code, not that this approach is more attractive. I think checksumming on content|index|both would reasonably work (both is probably better b/c it's all technically "content"), but logically we shouldn't be seeing the same checksums when summing the blooms if their metadata is different (outside of statistically unlikely collisions). Does that make sense? |
It sure does make sense. I tried to communicate in the last meeting that this was a cheap work around to unblock the team. Bc I could not find a bug in the checksum code, hence i offered this. I'll spend more time looking at the checksum code. But I don't think it harms to merge this while keeping the bug issue open. |
Thanks for addt'l context that this was just unblocking our current approach. A few bits in no particular order:
Let's spend some more time uncovering why we're seeing identical checksums before moving to this approach. |
@paul1r and I were debugging this issue and got to the bottom of the why. We calculate the checksum with BloomPageHeader, which does not contain any information about actual bloom data. All fields can turn out to be the same for same number of pages for the same number of series in a bloom. We now consider checksumming over both index and blooms OR adding a hash to the |
What this PR does / why we need it:
We noticed that blooms created for different fingerprint(FPs) ranges and timestamps(Ts) were returning same checksums.
Block building essentially creates blooms and indexes. Previously we were calculating checksums based on blooms. However they do not contain information about FP and Ts. Moving checksum calculation to indexes includes that information, and results in different checksums as expected.
Which issue(s) this PR fixes:
Fixes https://github.com/grafana/loki-private/issues/899
Special notes for your reviewer:
The tests assume the chunkRefs in series are the same, for simplicity I kept this as is. But tested locally with different chunks refs too.
Checklist
CONTRIBUTING.md
guide (required)CHANGELOG.md
updatedadd-to-release-notes
labeldocs/sources/setup/upgrade/_index.md
production/helm/loki/Chart.yaml
and updateproduction/helm/loki/CHANGELOG.md
andproduction/helm/loki/README.md
. Example PRdeprecated-config.yaml
anddeleted-config.yaml
files respectively in thetools/deprecated-config-checker
directory. Example PR