Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduced lock file to shuffle sharding grouper #4805

Merged
merged 20 commits into from
Sep 3, 2022
Merged

Introduced lock file to shuffle sharding grouper #4805

merged 20 commits into from
Sep 3, 2022

Conversation

alexqyle
Copy link
Contributor

@alexqyle alexqyle commented Jul 28, 2022 •
edited
Loading

What this PR does:

  • Introduced lock file for every blocks are under compaction while using shuffle sharding grouper. The content of lock file is simply a timestamp when the lock file is updated by heart beat process while under compaction. The lock file will make sure blocks are under compaction will not be picked up by other compactor at the same time which fixed Add shuffle-sharding for the compactor #4433 (comment)
  • Grouper will return the first group with all blocks not having a lock file or timestamp inside lock file is more than 5 minutes old. This replaced the logic using group hash to determine which compactor should own which group. After this change, every compactor can compact groups that belong to the tenant assigned to the compactor and the blocks in the group are not locked.
  • Incoming groups are still sorted in the same way that smallest range, oldest min time first. The reason only first group would return is that big tenant compaction would not block compaction for other tenant for a long period of time.

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@alexqyle alexqyle force-pushed the lock-file-for-cortex-compactor branch from 682fd23 to 248eab1 Compare July 28, 2022 01:23
@alexqyle alexqyle marked this pull request as ready for review July 28, 2022 01:28
@pull-request-size pull-request-size bot added size/XL and removed size/L labels Aug 2, 2022
alexqyle and others added 10 commits August 2, 2022 10:43
@alexqyle
Introduced lock file to shuffle sharding grouper
dc65d95 
Signed-off-by: Alex Le <leqiyue@amazon.com>
@mengmengy @alexqyle
let redis cache logs log with context (cortexproject#4785)
41e1f08 
* let redis cache logs log with context

Signed-off-by: Mengmeng Yang <mengmengyang616@gmail.com>

* fix import

Signed-off-by: Mengmeng Yang <mengmengyang616@gmail.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>
@danielblando @alexqyle
DoBatch preference to 4xx if error (cortexproject#4783)
20e6502 
* DoBatch preference to 4xx if error

Signed-off-by: Daniel Blando <ddeluigg@amazon.com>

* Fix comment

Signed-off-by: Daniel Blando <ddeluigg@amazon.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>
@alexqyle
Updated CHANGELOG and ordered imports
735e10d 
Signed-off-by: Alex Le <leqiyue@amazon.com>
@alexqyle
Fixed lint and removed groupCallLimit
a74e8de 
Signed-off-by: Alex Le <leqiyue@amazon.com>
@alexqyle
Changed lock file to json format and make sure planner would not pick…
91e381a 
… up group that is locked by other compactor

Signed-off-by: Alex Le <leqiyue@amazon.com>
@alanprot @alexqyle
Fix updateCachedShippedBlocks - new thanos (cortexproject#4806)
f45290c 
Signed-off-by: Alan Protasio <approtas@amazon.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>
@danielblando @alexqyle
Join memberlist on starting with no retry (cortexproject#4804)
9c3efbc 
Signed-off-by: Daniel Blando <ddeluigg@amazon.com>
@damnever @alexqyle
Fix alertmanager log message (cortexproject#4801)
275881a 
Signed-off-by: Xiaochao Dong (@damnever) <the.xcdong@gmail.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>
@alvinlin123 @alexqyle
Grafana Cloud uses Mimir now, so remove Grafana Cloud as hosted servi…
d6d9349 
…ce in documents (cortexproject#4809)

* Grafana Cloud uses Mimir, for of Cortex, now

Signed-off-by: Alvin Lin <alvinlin123@gmail.com>

* Improve doc

Signed-off-by: Alvin Lin <alvinlin@amazon.com>
Signed-off-by: Alex Le <leqiyue@amazon.com>
@alexqyle alexqyle force-pushed the lock-file-for-cortex-compactor branch from 9089ea3 to d6d9349 Compare August 2, 2022 17:50
@alexqyle
Merge branch 'master' into lock-file-for-cortex-compactor
4714e83 
Signed-off-by: Alex Le <emoc1989@gmail.com>
roystchiang
roystchiang reviewed Aug 3, 2022
View reviewed changes
pkg/compactor/shuffle_sharding_grouper.go Outdated Show resolved Hide resolved
pkg/compactor/shuffle_sharding_grouper.go Outdated Show resolved Hide resolved
pkg/compactor/shuffle_sharding_grouper.go Outdated Show resolved Hide resolved
pkg/compactor/shuffle_sharding_grouper.go Outdated Show resolved Hide resolved
pkg/compactor/shuffle_sharding_planner.go Outdated Show resolved Hide resolved
@alexqyle
Created block_locker to handle all block lock file operations. Added …
7cf4862 
…block lock metrics.

Signed-off-by: Alex Le <leqiyue@amazon.com>
roystchiang
roystchiang reviewed Aug 4, 2022
View reviewed changes
pkg/compactor/block_locker.go Outdated Show resolved Hide resolved
pkg/compactor/block_locker.go Outdated Show resolved Hide resolved
pkg/compactor/shuffle_sharding_planner.go Outdated Show resolved Hide resolved
pkg/compactor/shuffle_sharding_grouper.go Outdated Show resolved Hide resolved
alexqyle and others added 3 commits August 4, 2022 15:23
@alexqyle
Moved lock file heart beat into planner and refined planner logic to …
835251a 
…make sure blocks are locked by current compactor

Signed-off-by: Alex Le <leqiyue@amazon.com>
@alexqyle
Updated documents
3e5f65d 
Signed-off-by: Alex Le <leqiyue@amazon.com>
@alexqyle
Merge branch 'master' into lock-file-for-cortex-compactor
00ea6da 
Signed-off-by: Alex Le <emoc1989@gmail.com>
@alanprot
Merge branch 'master' into lock-file-for-cortex-compactor
a5a47b7 
Signed-off-by: Alan Protasio <approtas@amazon.com>
@alanprot
Copy link
Member

This is amazing overall! Just added a few nits!

@alexqyle
Return concurrency number of group. Use ticker for lock file heart beat
d353004 
Signed-off-by: Alex Le <leqiyue@amazon.com>
@alexqyle
Renamed lock file to be visit marker file
b7b0663 
Signed-off-by: Alex Le <leqiyue@amazon.com>
@alexqyle
Fixed unit test
ad5963b 
Signed-off-by: Alex Le <leqiyue@amazon.com>
@alexqyle
Make sure visited block can be picked by compactor visited it
c47eb5c 
Signed-off-by: Alex Le <leqiyue@amazon.com>
@alanprot alanprot merged commit 546d044 into cortexproject:master Sep 3, 2022
@alanprot alanprot mentioned this pull request Sep 19, 2022
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alanprot alanprot alanprot left review comments

@roystchiang roystchiang roystchiang left review comments

@alvinlin123 alvinlin123 alvinlin123 approved these changes

Assignees
No one assigned
Labels
size/XL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@alexqyle @alanprot @alvinlin123 @roystchiang @mengmengy @danielblando @damnever