Rollup indicator cleanup #1508

Dieterbe · 2019-10-24T21:01:00Z

main changes:

cleaner output for users
also include archive interval, because interval may not match what schema says. e.g. for kafka input plugin.

I need to fix the unit tests, but it works.
verified with fakemetrics to feed some data at 5s, and some at 1s + a storage-schemas.conf like so:

[default-1]
pattern = some.*1
retentions = 1s:6h:2min:2,1min:35d:6h:1

[default-2]
pattern = some.*2
retentions = 2s:6h:2min:2,1min:35d:6h:1

[default-4]
pattern = some.*4
retentions = 4s:6h:2min:2,1min:35d:6h:1

[default-5]
pattern = some.*5
retentions = 5s:6h:2min:2,1min:35d:6h:1

[default]
pattern = .*
retentions = 10s:6h:2min:2

curl 'http://localhost:6061/render?target=sum(some.id.of.a.metric.*)&from=-30s&meta=true' | jsonpp | less
{
    "version": "v0.1",
    "meta": {
        "stats": {
            "executeplan.resolve-series.ms": 8,
            "executeplan.get-targets.ms": 11,
            "executeplan.prepare-series.ms": 0,
            "executeplan.plan-run.ms": 0,
            "executeplan.series-fetch.count": 300,
            "executeplan.points-fetch.count": 6600,
            "executeplan.points-return.count": 1800,
            "executeplan.cache-miss.count": 0,
            "executeplan.cache-hit-partial.count": 0,
            "executeplan.cache-hit.count": 0,
            "executeplan.chunks-from-tank.count": 300,
            "executeplan.chunks-from-cache.count": 0,
            "executeplan.chunks-from-store.count": 0
        }
    },
    "series": [
        {
            "target": "sumSeries(some.id.of.a.metric.*)",
            "datapoints": [
                [
                    9990.44220170934,
                    1571948135
                ],
                [
                    10350.177267618516,
                    1571948140
                ],
                [
                    9887.481073889445,
                    1571948145
                ],
                [
                    10312.657377260079,
                    1571948150
                ],
                [
                    2473.5966041257966,
                    1571948155
                ],
                [
                    null,
                    1571948160
                ]
            ],
            "meta": [
                {
                    "schema-name": "default-1",
                    "schema-retentions": "1s:6h:2min:2,1min:35d:6h:1",
                    "archive-read": 0,
                    "archive-interval": 5,
                    "aggnum-norm": 1,
                    "consolidate-normfetch": "AverageConsolidator",
                    "aggnum-rc": 0,
                    "consolidate-rc": "NoneConsolidator",
                    "count": 20
                },
                {
                    "schema-name": "default-1",
                    "schema-retentions": "1s:6h:2min:2,1min:35d:6h:1",
                    "archive-read": 0,
                    "archive-interval": 1,
                    "aggnum-norm": 5,
                    "consolidate-normfetch": "AverageConsolidator",
                    "aggnum-rc": 0,
                    "consolidate-rc": "NoneConsolidator",
                    "count": 119
                },
                {
                    "schema-name": "default-2",
                    "schema-retentions": "2s:6h:2min:2,1min:35d:6h:1",
                    "archive-read": 0,
                    "archive-interval": 5,
                    "aggnum-norm": 1,
                    "consolidate-normfetch": "AverageConsolidator",
                    "aggnum-rc": 0,
                    "consolidate-rc": "NoneConsolidator",
                    "count": 17
                },
                {
                    "schema-name": "default-2",
                    "schema-retentions": "2s:6h:2min:2,1min:35d:6h:1",
                    "archive-read": 0,
                    "archive-interval": 1,
                    "aggnum-norm": 5,
                    "consolidate-normfetch": "AverageConsolidator",
                    "aggnum-rc": 0,
                    "consolidate-rc": "NoneConsolidator",
                    "count": 18
                },
                {
                    "schema-name": "default",
                    "schema-retentions": "10s:6h:2min:2",
                    "archive-read": 0,
                    "archive-interval": 5,
                    "aggnum-norm": 1,
                    "consolidate-normfetch": "AverageConsolidator",
                    "aggnum-rc": 0,
                    "consolidate-rc": "NoneConsolidator",
                    "count": 35
                },
                {
                    "schema-name": "default",
                    "schema-retentions": "10s:6h:2min:2",
                    "archive-read": 0,
                    "archive-interval": 1,
                    "aggnum-norm": 5,
                    "consolidate-normfetch": "AverageConsolidator",
                    "aggnum-rc": 0,
                    "consolidate-rc": "NoneConsolidator",
                    "count": 35
                },
                {
                    "schema-name": "default-4",
                    "schema-retentions": "4s:6h:2min:2,1min:35d:6h:1",
                    "archive-read": 0,
                    "archive-interval": 5,
                    "aggnum-norm": 1,
                    "consolidate-normfetch": "AverageConsolidator",
                    "aggnum-rc": 0,
                    "consolidate-rc": "NoneConsolidator",
                    "count": 15
                },
                {
                    "schema-name": "default-4",
                    "schema-retentions": "4s:6h:2min:2,1min:35d:6h:1",
                    "archive-read": 0,
                    "archive-interval": 1,
                    "aggnum-norm": 5,
                    "consolidate-normfetch": "AverageConsolidator",
                    "aggnum-rc": 0,
                    "consolidate-rc": "NoneConsolidator",
                    "count": 15
                },
                {
                    "schema-name": "default-5",
                    "schema-retentions": "5s:6h:2min:2,1min:35d:6h:1",
                    "archive-read": 0,
                    "archive-interval": 5,
                    "aggnum-norm": 1,
                    "consolidate-normfetch": "AverageConsolidator",
                    "aggnum-rc": 0,
                    "consolidate-rc": "NoneConsolidator",
                    "count": 13
                },
                {
                    "schema-name": "default-5",
                    "schema-retentions": "5s:6h:2min:2,1min:35d:6h:1",
                    "archive-read": 0,
                    "archive-interval": 1,
                    "aggnum-norm": 5,
                    "consolidate-normfetch": "AverageConsolidator",
                    "aggnum-rc": 0,
                    "consolidate-rc": "NoneConsolidator",
                    "count": 13
                }
            ]
        }
    ]
}

This will become useful in response metadata when we want to include theschema used

in particular, show the actual schema used

e.g. kafka input

Dieterbe · 2019-10-28T08:05:59Z

fix #1486

conf/retention_test.go

mdata/aggregator.go

replay

LGTM with one comment

I broke the build in #1508

Dieterbe added 2 commits October 25, 2019 11:44

Retentions: add field to track original string input

d5f32fd

This will become useful in response metadata when we want to include theschema used

comments

388b2bb

Dieterbe force-pushed the rollup-indicator-cleanup branch from e05f9dc to 08945be Compare October 25, 2019 10:05

Dieterbe added 6 commits October 25, 2019 17:17

return rollup info in a more user-friendly way

cff0862

in particular, show the actual schema used

also track archinterval because interval may not match the schema!

200eb1f

e.g. kafka input

go generate ./...

155347a

fix tests

03fdb24

simplify

0169a31

update http docs for meta=true

71fe346

Dieterbe force-pushed the rollup-indicator-cleanup branch from 08945be to 54b7fb2 Compare October 25, 2019 22:28

Dieterbe added 3 commits October 26, 2019 01:02

better way to fix tests

687c014

tweak test config to be actually valid

45a701f

unit test for buildOrigFromRetentions

fca78e1

Dieterbe force-pushed the rollup-indicator-cleanup branch from 54b7fb2 to fca78e1 Compare October 25, 2019 23:02

Dieterbe changed the title ~~WIP Rollup indicator cleanup~~ Rollup indicator cleanup Oct 25, 2019

replay reviewed Oct 28, 2019

View reviewed changes

conf/retention_test.go Show resolved Hide resolved

replay reviewed Oct 28, 2019

View reviewed changes

mdata/aggregator.go Outdated Show resolved Hide resolved

replay approved these changes Oct 28, 2019

View reviewed changes

pass retention orig properly in NewAggregator

849fa01

Dieterbe merged commit 38028b3 into master Nov 4, 2019

Dieterbe deleted the rollup-indicator-cleanup branch November 4, 2019 13:56

Dieterbe added a commit that referenced this pull request Nov 4, 2019

fix build.

367c1bc

I broke the build in #1508

Dieterbe mentioned this pull request Nov 4, 2019

fix build. #1519

Merged

Dieterbe mentioned this pull request Jan 22, 2020

Fix Retention.Sub() Orig string cutting #1624

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rollup indicator cleanup #1508

Rollup indicator cleanup #1508

Dieterbe commented Oct 24, 2019

Dieterbe commented Oct 28, 2019

replay left a comment

Rollup indicator cleanup #1508

Rollup indicator cleanup #1508

Conversation

Dieterbe commented Oct 24, 2019

Dieterbe commented Oct 28, 2019

replay left a comment

Choose a reason for hiding this comment