introduce lastSave and adjust lastUpdate checks:

[Dieterbe]

we now use:

• lastSave = last save TS
• lastUpdate = last point seen

using lastUpdate for everything like we did causes issues for
backfills (#568)
and other scenarios where the data TS is offset against wall clock.

index filtering -> compare lastUpdate to from
index pruning -> checks lastSave
cassandra updating -> checks lastSave

[replay]

This seems to rely on schema.MetricDefinition to get a new field LastSave. I know that this is maintained by Raintank, but can we just add a field there? Do we need to update it's version number if we do that? Or do you plan to only keep this change in our vendor copy of it and not update upstream?

[replay]

Do we actually need to save LastSave into cassandra? Wouldn't it be sufficient to just set it to LastUpdate on load?

[replay]

@Dieterbe If we already know at this point that the metric will be rejected once we try adding it, shouldn't we just return an error here and then make the input.DefaultHandler.Process() abort instead of letting it continue?

[Dieterbe]

that's a good suggestion. though note that right now there's an important metric tank.metrics_too_old being incremented in the AggMetric.Add code. it's important we communicate this back to the user. i suppose we could increment that metric from here and then do as you say. (this would also bypass the usage tracking for these points - I think we should account for all points even if we reject them - though IIRC we plan to move the usage tracking to tsdb-gw so I think that's ok)

[woodsaj]

existing.LastUpdate will be greater then def.LastUpdate when processing a backlog at startup. We need to accept those metrics to refill the aggMetric chunks

[replay]

Just to make sure i understand that right... if we're processing the backlog then MT will come up, read the index from cassandra, and then read all the insertions from kafka and apply them. if other nodes have already updated the cassandra index then the LastUpdate property will have been set by them and that's why it will be ahead of the data coming in from Kafka, right?
Good point...

[woodsaj]

• metrictank writes update to cassandra with LastUpdate = 10
• metrictank is restarted
• metrictank loads the index from cassandra, lastUpdate=10
• metrictank starts ingesting metrics from 6hours ago.
• first point seen has Time=0, so MetricDefinitionFromMetricData(data) will set LastUpdate to 0.
• existing.LastUpdate=10, def.LastUpdate=0. 10 > 0

[replay]

K, so I'll just leave that as it is. Then the metric should still be processed even though its LastUpdate is < existing.LastUpdate.