Remove tenant when last block is removed from local backend #1947
Conversation
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
joe-elliott
requested review from
annanay25,
mdisibio,
mapno,
yvrhdn and
zalegrala
as code owners
|
Another note: Currently when rediscovering local blocks this PR will nicely skip tenants that don't have blocks. However, it will leave those folders forever. Only when a block is cleared from the local backend do we make the check on whether or not to remove the tenant. This PR, as is, will not clean up already existing empty tenant folders in the local backend. |
Does it make sense to do local folder cleanup at ingester shutdown, which could also handle this scenario? |
This is likely the better approach. Let me see if I can put something together 👍 |
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
|
@mdisibio i took your suggestion and implemented clean up on shutdown. let me know what you think |
Signed-off-by: Joe Elliott <number101010@gmail.com>
| @@ -161,9 +161,27 @@ func (rw *Backend) ReadRange(ctx context.Context, name string, keypath backend.K | |||
| return nil | |||
| } | |||
|
|
|||
| // Shutdown implements backend.Reader | |||
| // Shutdown implements backend.Reader. It attempts to clear all tenants | |||
| // that do not have blocks. | |||
| func (rw *Backend) Shutdown() { | |||
There was a problem hiding this comment.
Forgot this existed. Excellent.
There was a problem hiding this comment.
Yeah. It's weird that Shutdown() on the Reader interface does the job of cleaning up, but meh. We should combine these interfaces anyway. I don't think much is gained from the split up.
(or just add Shutdown to the writer?)
What this PR does:
This fixes a long running issue where tenants were persisted even if they had a zero length wal. The issue was that when rediscovering local blocks a tenant was still found even if there were no blocks. This PR fixes the issue with two changes:
Technically only the first of the two is needed and would be a safer change. Having the second is nice because it will clear up folders in our ingester local backends. Should we remove the second change and err towards safety? Especially with the 2.0 release coming up?
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]