Add proposal for parallel compaction by time interval #4272
Conversation
roystchiang
force-pushed
the
compactor-proposal
branch
from
d94b281
to
b386d7b
Compare
|
|
||
| ### Bad block resulting in non-ideal compaction groups | ||
|
|
||
| A Cortex operator configures the compaction block range. Using 2h and 6h as example, [2h-1] [2h-2] [2h-3] [2h-4] [2h-5] [2h-6]. If the [2h-1] block is corrupted, we may compact the subsequent [2h-2] [2h-3] [2h-4] [2h-5] [2h-6] blocks. To compact into a 6 hour group, the ideal compaction is [2h-1] [2h-2] [2h-3] and [2h-4] [2h-5] [2h-6]. The cortex planner needs to know the ideal compaction interval, and prevent compaction of [2h-2] [2h-3] [2h-4] from happening, which will result in [2h-1] not able to be compacted into longer time interval blocks. Cortex has full information regarding all the available blocks, so we should utilize this information to achieve the best compaction interval. |
There was a problem hiding this comment.
I didn't follow this. What sort of corruption are you thinking about?
Is it related to this? thanos-io/thanos#4046
There was a problem hiding this comment.
I'm thinking of #3569
I suppose it's not exactly a corruption, but something that prevents compactor from compacting currently.
There was a problem hiding this comment.
I'm not sure I fully understand the proposed approach here. You can still compact all non-corrupted blocks together and just leave out of the compaction the corrupted block.
There was a problem hiding this comment.
I did not want to introduce a different behavior from what is currently happening. I wanted the proposal to be scoped to introducing horizontal scaling, and we can work on skipping bad blocks at a later stage.
There was a problem hiding this comment.
That idea seems fine. The amount of detail in the example seems overkill to make the point.
| On resharding of compactor schedulers, a tenant might move to a different scheduler. We can either drop the current compactor job in order to prevent duplicate compaction jobs, or continue compaction. I propose that the compactor drops the compaction job if the compaction group no longer belongs to the original compactor scheduler. This way, we do not have duplicate compactions happening, and we can minimize work wasted. | ||
|
|
||
| ### Contribute to Thanos for a more scalable compactor | ||
| Instead of introducing parallelism on the Cortex compactor level, we move the parallelism to the Thanos compactor itself. Thanos has a [proposal to make compactor more scalable](https://docs.google.com/document/d/1xi0V8DB0hE54XgkogJRnNL6yH7C5JThJywlLFoC6dCQ/), and a [PR](https://github.com/thanos-io/thanos/pull/3807). Cortex will enjoy higher throughput per tenant if Thanos is able to speed up the compaction, and we can keep the Cortex architecture the same. However, this approach means that a single tenant is still sharded to a single compactor. In order to compact more groups at once, we must scale up compactor vertically. Although vertical scaling can get us far, we should scale horizontally where we can. |
There was a problem hiding this comment.
That Thanos PR seems to be stalled; looks like Bartek called for a redesign.
The Google doc is just laying out the goals; it just says "TBD" for what would actually be done.
There was a problem hiding this comment.
Do you think it still makes sense to reference it given that it is not really actively being developed?
There was a problem hiding this comment.
Definitely, always good to see where others have been before.
I guess I mostly wanted to clarify that this isn't something we can build on directly.
|
|
||
| ### Bad block resulting in non-ideal compaction groups | ||
|
|
||
| A Cortex operator configures the compaction block range. Using 2h and 6h as example, [2h-1] [2h-2] [2h-3] [2h-4] [2h-5] [2h-6]. If the [2h-1] block is corrupted, we may compact the subsequent [2h-2] [2h-3] [2h-4] [2h-5] [2h-6] blocks. To compact into a 6 hour group, the ideal compaction is [2h-1] [2h-2] [2h-3] and [2h-4] [2h-5] [2h-6]. The cortex planner needs to know the ideal compaction interval, and prevent compaction of [2h-2] [2h-3] [2h-4] from happening, which will result in [2h-1] not able to be compacted into longer time interval blocks. Cortex has full information regarding all the available blocks, so we should utilize this information to achieve the best compaction interval. |
There was a problem hiding this comment.
I'm not sure I fully understand the proposed approach here. You can still compact all non-corrupted blocks together and just leave out of the compaction the corrupted block.
Signed-off-by: Roy Chiang <roychi@amazon.com>
Signed-off-by: Roy Chiang <roychi@amazon.com>
roystchiang
force-pushed
the
compactor-proposal
branch
from
13855cd
to
b04d90f
Compare
|
|
||
| ### Bad block resulting in non-ideal compaction groups | ||
|
|
||
| A Cortex operator configures the compaction block range. Using 2h and 6h as example, [2h-1] [2h-2] [2h-3] [2h-4] [2h-5] [2h-6]. If the [2h-1] block is corrupted, we may compact the subsequent [2h-2] [2h-3] [2h-4] [2h-5] [2h-6] blocks. To compact into a 6 hour group, the ideal compaction is [2h-1] [2h-2] [2h-3] and [2h-4] [2h-5] [2h-6]. The cortex planner needs to know the ideal compaction interval, and prevent compaction of [2h-2] [2h-3] [2h-4] from happening, which will result in [2h-1] not able to be compacted into longer time interval blocks. Cortex has full information regarding all the available blocks, so we should utilize this information to achieve the best compaction interval. |
There was a problem hiding this comment.
That idea seems fine. The amount of detail in the example seems overkill to make the point.
| On resharding of compactor schedulers, a tenant might move to a different scheduler. We can either drop the current compactor job in order to prevent duplicate compaction jobs, or continue compaction. I propose that the compactor drops the compaction job if the compaction group no longer belongs to the original compactor scheduler. This way, we do not have duplicate compactions happening, and we can minimize work wasted. | ||
|
|
||
| ### Contribute to Thanos for a more scalable compactor | ||
| Instead of introducing parallelism on the Cortex compactor level, we move the parallelism to the Thanos compactor itself. Thanos has a [proposal to make compactor more scalable](https://docs.google.com/document/d/1xi0V8DB0hE54XgkogJRnNL6yH7C5JThJywlLFoC6dCQ/), and a [PR](https://github.com/thanos-io/thanos/pull/3807). Cortex will enjoy higher throughput per tenant if Thanos is able to speed up the compaction, and we can keep the Cortex architecture the same. However, this approach means that a single tenant is still sharded to a single compactor. In order to compact more groups at once, we must scale up compactor vertically. Although vertical scaling can get us far, we should scale horizontally where we can. |
There was a problem hiding this comment.
Definitely, always good to see where others have been before.
I guess I mostly wanted to clarify that this isn't something we can build on directly.
| --- | ||
|
|
||
| ## Introduction | ||
| As a part of pushing Cortex’s scaling capability at AWS, we have done performance testing with Cortex and found the compactor to be one of the main limiting factors for higher active timeseries limit per tenant. The documentation [Compactor](https://cortexmetrics.io/docs/blocks-storage/compactor/#how-compaction-works) describes the responsibilities of a compactor, and this proposal focuses on the limitations of the current compactor architecture. In the current architecture, compactor has simple sharding, meaning that a single tenant is sharded to a single compactor. In addition, a compactor handles compaction groups of a single tenant iteratively, meaning that blocks belonging non-overlapping times are not compacted in parallel. |
There was a problem hiding this comment.
"compaction groups" is not defined before use, either in this document or at the referenced link.
Possibly it could go in https://github.com/cortexproject/cortex/blob/master/docs/guides/glossary.md
| As a part of pushing Cortex’s scaling capability at AWS, we have done performance testing with Cortex and found the compactor to be one of the main limiting factors for higher active timeseries limit per tenant. The documentation [Compactor](https://cortexmetrics.io/docs/blocks-storage/compactor/#how-compaction-works) describes the responsibilities of a compactor, and this proposal focuses on the limitations of the current compactor architecture. In the current architecture, compactor has simple sharding, meaning that a single tenant is sharded to a single compactor. In addition, a compactor handles compaction groups of a single tenant iteratively, meaning that blocks belonging non-overlapping times are not compacted in parallel. | ||
|
|
||
| ### Problem and Requirements | ||
| Currently, a compactor is able to compact up to 20M timeseries within 2 hours for a level-2 compaction, including the time to download blocks, compact, and upload the newly compacted block. We would like to increase the timeseries limit per tenant, and compaction is one of the limiting factors. In addition, we would like to achieve the following: |
There was a problem hiding this comment.
I cannot find a definition of "level-2" here or in Compactor docs either.
|
I'm going to merge the proposal to move forward but I would be glad if you could address Bryan comments in a follow up PR. Thanks! |
|
@roystchiang gentle reminder I made some points about explaining terms, and I couldn't see any follow-up. |
I will submit a follow up PR this week. Thanks for the reminder. |
What this PR does:
Add proposal for parallel compaction by time interval
Which issue(s) this PR fixes:
Proposal to work on a fix for #3753
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]