New flag 'ingest-from'

[fkaleo]

New flag 'ingest-from' that instructs metrictank to discard any point of a specific org id that does not belong to chunks starting after a given timestamp.
e.g. 20:67031 with a chunk span of 10 will discard all points with org id 20 and timestamps lesser than 67040

[woodsaj]

why limit this to only 1 orgId? Whould it not be better to just have AggMetrics use a map of orgId's to ingestAfter timstamp. eg ingestAfter map[int32]int32

Users can then pass a list of org:ts pairs via the config. eg
metrictank --ingest-after=1:1562339919,1234:1562339919,45:1562339919

[fkaleo]

why limit this to only 1 orgId? Whould it not be better to just have AggMetrics use a map of orgId's to ingestAfter timstamp. eg ingestAfter map[int32]int32

Users can then pass a list of org:ts pairs via the config. eg
metrictank --ingest-after=1:1562339919,1234:1562339919,45:1562339919

The only reason was the fact that we did not need it and kept the code a little simpler. We can always add it later.

[fkaleo]

why limit this to only 1 orgId? Whould it not be better to just have AggMetrics use a map of orgId's to ingestAfter timstamp. eg ingestAfter map[int32]int32
Users can then pass a list of org:ts pairs via the config. eg
metrictank --ingest-after=1:1562339919,1234:1562339919,45:1562339919

The only reason was the fact that we did not need it and kept the code a little simpler. We can always add it later.

Using a map is probably less code though; let me try that.

[robert-milan]

I think this might need to be rebased on master

[fkaleo]

I think this might need to be rebased on master

Done

[Dieterbe]

too tired now, but tomorrow i'ld like to look into this one last thing, about the timestamps going 130->120 and how that may or should not affect things.

[fkaleo]

LGTM. just need to look at one last thing tomorrow.

Thank you! Your changes look great.

[Dieterbe]

LGTM. just need to look at one last thing tomorrow.

[Dieterbe]

LGTM
Please have a close look at the last 2 commits I added.
If they make sense, merge this.
If not, let's talk.

[fkaleo]

LGTM
Please have a close look at the last 2 commits I added.
If they make sense, merge this.
If not, let's talk.

They make sense. I'm adding test cases to cover the last one.

[Dieterbe]

I already covered that in the test case i added.
if we need any more tests, it should be one that uses an actual AggMetric with aggregators on it, and confirms the right points make it in to the aggregator (points in the last aggspan before the first t0)

[fkaleo]

I already covered that in the test case i added.
if we need any more tests, it should be one that uses an actual AggMetric with aggregators on it, and confirms the right points make it in to the aggregator (points in the last aggspan before the first t0)

The test case added does not fail when removing the code changes.
And yep, I'm adding a test to an AggMetric and discovering that in fact more code changes need to happen to make it pass, I think.

[fkaleo]

Please take a look at the 2 additional tests and the change in code and let me know what you think.