Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blooms: Builder can delete blocks that are referenced in up-to-date metas #13612

Closed
chaudum opened this issue Jul 22, 2024 · 0 comments · Fixed by #13614
Closed

Blooms: Builder can delete blocks that are referenced in up-to-date metas #13612

chaudum opened this issue Jul 22, 2024 · 0 comments · Fixed by #13614
Labels
feature/blooms type/bug Somehing is not working as expected

Comments

@chaudum
Copy link
Contributor

chaudum commented Jul 22, 2024

Describe the bug

It occurs that the builder (and likely the compactor as well) delete blocks that are referenced in up-to-date metas in case there haven't been any changes to the block between different versions of the TSDB that has been used to build the block.

This is, because the new meta which is created from the block and the new TSDB is uploaded alongside the existing block that has already been uploaded before in a previous iteration.
However, since the planner also deletes the blocks referenced in out-of-date metas, the block is removed from storage, even though it's also references in the new meta.

This behaviour causes the gateway to not find these blocks in storage yielding to a low cache hit rate and lower filter rate.

Example logs from two tasks operating on the same keyspace:

2024-07-22 04:14:02.933	99.97 | bloom/loki_ops_tsdb_index_19924/29/blocks/c26612af1b6db20a-c27ffbbfa9788f29/1721399531187-1721584994856-5955b215.tar.gz | 1721609358792956006-compactor-1721384279065-1721608391504-d6cdc909.tsdb | loki_ops_tsdb_index_19924-29-c23ffffffffffcf7-c27ffffffffffcf5-3603810569-1
2024-07-22 04:13:55.733	59.45 | bloom/loki_ops_tsdb_index_19924/29/blocks/c252264bbef1ef64-c2660c8fe79bba98/1721398483206-1721583667512-44e86c07.tar.gz | 1721609358792956006-compactor-1721384279065-1721608391504-d6cdc909.tsdb | loki_ops_tsdb_index_19924-29-c23ffffffffffcf7-c27ffffffffffcf5-3603810569-1
2024-07-22 04:13:47.033	28.28 | bloom/loki_ops_tsdb_index_19924/29/blocks/c2400835b3183e9a-c25219830390a7f1/1721399777836-1721585439219-775fa136.tar.gz | 1721609358792956006-compactor-1721384279065-1721608391504-d6cdc909.tsdb | loki_ops_tsdb_index_19924-29-c23ffffffffffcf7-c27ffffffffffcf5-3603810569-1
2024-07-21 22:59:24.199	99.97 | bloom/loki_ops_tsdb_index_19924/29/blocks/c26612af1b6db20a-c27ffbbfa9788f29/1721399531187-1721584994856-5955b215.tar.gz | 1721594936833552908-compactor-1721384279065-1721594045905-894a946.tsdb | loki_ops_tsdb_index_19924-29-c23ffffffffffcf7-c27ffffffffffcf5-143960390-1
2024-07-21 22:57:05.099	59.45 | bloom/loki_ops_tsdb_index_19924/29/blocks/c252264bbef1ef64-c2660c8fe79bba98/1721398483206-1721583667512-44e86c07.tar.gz | 1721594936833552908-compactor-1721384279065-1721594045905-894a946.tsdb | loki_ops_tsdb_index_19924-29-c23ffffffffffcf7-c27ffffffffffcf5-143960390-1
2024-07-21 22:44:38.000	28.28 | bloom/loki_ops_tsdb_index_19924/29/blocks/c2400835b3183e9a-c25219830390a7f1/1721399777836-1721585439219-775fa136.tar.gz | 1721594936833552908-compactor-1721384279065-1721594045905-894a946.tsdb | loki_ops_tsdb_index_19924-29-c23ffffffffffcf7-c27ffffffffffcf5-143960390-1

Expected behavior
Blocks that are referenced in up-to-date metas are not deleted from storage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature/blooms type/bug Somehing is not working as expected
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant