Ruler: per-tenant WAL #4344
Merged
Ruler: per-tenant WAL #4344
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dannykopping
force-pushed
the
dannykopping/ruler-wal
branch
3 times, most recently
from
e161d53
to
104b810
Compare
dannykopping
commented
Sep 20, 2021
| @@ -154,7 +173,6 @@ func (l *Limits) RegisterFlags(f *flag.FlagSet) { | |||
|
|
|||
| f.IntVar(&l.RulerMaxRulesPerRuleGroup, "ruler.max-rules-per-rule-group", 0, "Maximum number of rules per rule group per-tenant. 0 to disable.") | |||
| f.IntVar(&l.RulerMaxRuleGroupsPerTenant, "ruler.max-rule-groups-per-tenant", 0, "Maximum number of rule groups per-tenant. 0 to disable.") | |||
| f.IntVar(&l.RulerRemoteWriteQueueCapacity, "ruler.remote-write.queue-capacity", 10000, "Capacity of remote-write queues; if a queue exceeds its capacity it will evict oldest samples.") | |||
There was a problem hiding this comment.
Note to self: add notice in changelog about this flag being removed
dannykopping
force-pushed
the
dannykopping/ruler-wal
branch
from
6652e52
to
8eae867
Compare
2 tasks
dannykopping
force-pushed
the
dannykopping/ruler-wal
branch
from
8eae867
to
f5d17b1
Compare
added 18 commits
September 20, 2021 15:12
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Setting WAL dir name as tenant ID Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Adding appender_ready metric Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
…rule evaluation until WAL is ready Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
added 17 commits
September 20, 2021 15:29
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
…p was removed in Prometheus Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
No reason to keep a per-tenant WAL without remote-write Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
We need to prevent a package import cycle Also add Clone functions for deep copy Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Clean up WAL dir after test Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
…t be manipulated Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
dannykopping
force-pushed
the
dannykopping/ruler-wal
branch
from
f5d17b1
to
363fd7f
Compare
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
dannykopping
force-pushed
the
dannykopping/ruler-wal
branch
from
8dadf6c
to
f48dd3b
Compare
It will only clean abandoned WALs on startup, for now Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
owen-d
approved these changes
Sep 21, 2021
|
|
||
| return ruler.ManagerFactory(func( | ||
| var registry storageRegistry |
There was a problem hiding this comment.
I don't love having this globally defined, perhaps we can alter it.
There was a problem hiding this comment.
I completely agree, but I don't see an easy way to do so.
I'll try address this in a follow-up PR.
|
|
||
| mgr := rules.NewManager(&rules.ManagerOptions{ | ||
| Appendable: newAppendable(cfg, overrides, logger, userID, rwMetrics), | ||
| Appendable: registry, | ||
| Queryable: memStore, |
There was a problem hiding this comment.
We're still using the memStore here as queryable, which is fine, but we are now able to read from Prometheus to determine this with less hacks and remove the memstore implementation altogether. This can be in a later PR though. I opened #4362
dannykopping
pushed a commit
that referenced
this pull request
Sep 27, 2021
Signed-off-by: Danny Kopping <danny.kopping@grafana.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Adds durability and isolation to the samples generated by Loki recording rules.
Each tenant will have their own Write-Ahead Log (WAL). Samples appended to the WAL will be sent to the configured remote-write endpoint separately for each tenant to prevent noisy neighbours.
Almost all remote-write configurations are configurable on a per-tenant basis, with each tenant defaulting to the global remote-write config. Remote-write can be reconfigured at runtime, too.
Which issue(s) this PR fixes:
Fixes #4241
Special notes for your reviewer:
We reuse a lot of Prometheus' WAL and remote-write functionality, as well as some code from the Grafana Agent. The Agent's WAL handling does away with the TSDB head appender; by skipping the head appender, we only need to keep a WAL per tenant, and not an additional tsdb.
The code from the Agent is heavily modified for our purposes.
Most of the changes I made in
pkg/ruler/storage(copied from grafana/agent in #4354) was around making metrics injectable, removing features we didn't need and minor refactoring.Checklist