This repository was archived by the owner on Aug 23, 2023. It is now read-only.
historical data is lost after a restart when the series interval has changed #813
Milestone
Comments
Aergonus
pushed a commit
to bloomberg/metrictank
that referenced
this issue
Jan 19, 2018
modifies the index loading so only names/tagSets are pruned where all MetricDefinitions are Stale. fixes grafana#813
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
f06e1fb changed how we handling deleting series. We now keep them in cassandra and just filter at startup.
This breaks some important logic that ensures previous versions of a series that are still active are not deleted. eg, if the interval of a series is changed a new metricDefinition is created. The series now has 2 metricDefs. The pruning logic in idx.Memory ensures that as long as one of these metricDefs is being updated both will be kept. Without this logic the historical data sent with the old interval will be lost.
We should be able to restore the previous logic by
removing https://github.com/grafana/metrictank/blob/master/idx/cassandra/cassandra.go#L363-L365
and updating https://github.com/grafana/metrictank/blob/master/idx/cassandra/cassandra.go#L349-L352 to scan the metricDefs and only remove defs if all versions for a series are older then maxStale
The text was updated successfully, but these errors were encountered: