[processor/lsminterval] Define cardinality limits and handle overflows

[lahsivjar]

Related to: #141

For more details checkout #141 (comment) comment.

[axw]

Doesn't this mean we risk OOM due to high cardinality metric names? Should the limit be on time series (metric + dimensions) per scope, rather than data points per scope? Or metrics per scope and time series per metric?

[axw]

At a high level I think the approach looks good. As mentioned in a comment, I think we should have limits on all parts of the hierarchy, including metrics.

I ran the benchmarks, and it seems there's quite a drop in throughput.

[lahsivjar]

I ran the benchmarks, and it seems there's quite a drop in throughput.

Yeah, there is a considerable overhead. This is what I am working on right now -- to improve performance and drop some of the unneeded complexity around attribute matching and filtering.

[lahsivjar]

@axw I have addressed all of the comments other than one and I am marking this ready for review now. The performance consideration is still there and I have a few threads to chase down for improvements but I expect them to give minor gains, so, if you have any ideas (even rough) I will be more than happy to chase them down. One of the sources of the extra overhead is creating the lookup maps on unmarshaling the binary to the go struct -- previously I was doing this when required but now I am always doing this to simplify overflow handling a bit.

I have left adding metrics limits to a follow-up PR. I will also add a few more detailed overflow tests.

[axw]

Preliminary comments, ran out of time today

[axw]

Looks good overall, mostly minor comments/questions

[lahsivjar]

Bringing this back to draft as I have a few good ideas for optimization and combined with @axw 's above comments I have a few more things to do here.

(I have also pushed a different way to encode limit trackers independent of the pmetric ds)

[lahsivjar]

After the above changes, below is the benchmark diff from main:

Benchmark diff from `main`

name                                            old time/op    new time/op    delta
Aggregation/sum_cumulative-10                     10.0µs ± 5%    12.2µs ± 5%  +22.80%  (p=0.008 n=5+5)
Aggregation/sum_delta-10                          10.0µs ± 3%    12.3µs ± 2%  +23.51%  (p=0.008 n=5+5)
Aggregation/histogram_cumulative-10               10.1µs ± 3%    12.8µs ± 4%  +26.10%  (p=0.008 n=5+5)
Aggregation/histogram_delta-10                    10.4µs ± 2%    12.9µs ± 2%  +24.41%  (p=0.008 n=5+5)
Aggregation/exphistogram_cumulative-10            11.1µs ± 2%    12.7µs ± 1%  +14.86%  (p=0.008 n=5+5)
Aggregation/exphistogram_delta-10                 11.2µs ± 2%    13.4µs ± 8%  +19.06%  (p=0.008 n=5+5)
Aggregation/summary_enabled-10                    10.3µs ± 1%    13.2µs ± 4%  +28.69%  (p=0.008 n=5+5)
Aggregation/summary_passthrough-10                1.30µs ± 2%    1.39µs ± 6%   +6.82%  (p=0.048 n=5+5)
AggregationWithOTTL/sum_cumulative-10             12.0µs ± 3%    14.4µs ± 6%  +19.39%  (p=0.008 n=5+5)
AggregationWithOTTL/sum_delta-10                  12.2µs ± 4%    14.3µs ± 4%  +16.75%  (p=0.008 n=5+5)
AggregationWithOTTL/histogram_cumulative-10       12.5µs ± 0%    14.4µs ± 7%  +15.09%  (p=0.016 n=4+5)
AggregationWithOTTL/histogram_delta-10            11.8µs ± 7%    15.2µs ± 6%  +28.66%  (p=0.008 n=5+5)
AggregationWithOTTL/exphistogram_cumulative-10    13.2µs ± 6%    14.3µs ± 5%   +8.42%  (p=0.016 n=5+5)
AggregationWithOTTL/exphistogram_delta-10         13.2µs ± 4%    14.9µs ± 1%  +13.16%  (p=0.008 n=5+5)
AggregationWithOTTL/summary_enabled-10            12.7µs ± 5%    14.1µs ± 3%  +11.16%  (p=0.008 n=5+5)
AggregationWithOTTL/summary_passthrough-10        1.40µs ± 5%    1.32µs ± 3%   -5.87%  (p=0.032 n=5+5)

name                                            old alloc/op   new alloc/op   delta
Aggregation/sum_cumulative-10                     17.6kB ± 3%    23.8kB ± 1%  +35.18%  (p=0.016 n=5+4)
Aggregation/sum_delta-10                          17.6kB ± 1%    23.9kB ± 1%  +35.23%  (p=0.008 n=5+5)
Aggregation/histogram_cumulative-10               19.4kB ± 2%    24.7kB ± 5%  +27.10%  (p=0.008 n=5+5)
Aggregation/histogram_delta-10                    19.4kB ± 1%    24.7kB ± 5%  +27.42%  (p=0.008 n=5+5)
Aggregation/exphistogram_cumulative-10            21.7kB ± 0%    24.2kB ± 1%  +11.70%  (p=0.008 n=5+5)
Aggregation/exphistogram_delta-10                 22.2kB ± 1%    24.9kB ± 1%  +12.11%  (p=0.008 n=5+5)
Aggregation/summary_enabled-10                    18.5kB ± 1%    21.9kB ± 4%  +18.36%  (p=0.016 n=4+5)
Aggregation/summary_passthrough-10                1.38kB ± 0%    1.38kB ± 0%   +0.52%  (p=0.032 n=5+5)
AggregationWithOTTL/sum_cumulative-10             20.2kB ± 1%    23.6kB ± 2%  +16.63%  (p=0.008 n=5+5)
AggregationWithOTTL/sum_delta-10                  20.2kB ± 1%    23.7kB ± 1%  +17.35%  (p=0.008 n=5+5)
AggregationWithOTTL/histogram_cumulative-10       22.1kB ± 3%    26.9kB ± 2%  +21.82%  (p=0.008 n=5+5)
AggregationWithOTTL/histogram_delta-10            21.9kB ± 2%    26.5kB ± 0%  +20.91%  (p=0.008 n=5+5)
AggregationWithOTTL/exphistogram_cumulative-10    23.0kB ± 5%    26.8kB ± 1%  +16.58%  (p=0.008 n=5+5)
AggregationWithOTTL/exphistogram_delta-10         23.6kB ± 3%    27.5kB ± 1%  +16.38%  (p=0.008 n=5+5)
AggregationWithOTTL/summary_enabled-10            20.7kB ± 4%    24.3kB ± 2%  +17.67%  (p=0.008 n=5+5)
AggregationWithOTTL/summary_passthrough-10        1.38kB ± 1%    1.38kB ± 0%     ~     (p=0.159 n=5+5)

name                                            old allocs/op  new allocs/op  delta
Aggregation/sum_cumulative-10                        199 ± 1%       235 ± 0%  +18.41%  (p=0.008 n=5+5)
Aggregation/sum_delta-10                             200 ± 0%       237 ± 0%  +18.06%  (p=0.008 n=5+5)
Aggregation/histogram_cumulative-10                  211 ± 1%       246 ± 0%  +16.67%  (p=0.008 n=5+5)
Aggregation/histogram_delta-10                       213 ± 0%       248 ± 0%  +16.34%  (p=0.008 n=5+5)
Aggregation/exphistogram_cumulative-10               228 ± 0%       264 ± 0%  +15.79%  (p=0.008 n=5+5)
Aggregation/exphistogram_delta-10                    237 ± 0%       273 ± 0%  +15.19%  (p=0.016 n=5+4)
Aggregation/summary_enabled-10                       212 ± 0%       248 ± 0%  +16.98%  (p=0.029 n=4+4)
Aggregation/summary_passthrough-10                  37.0 ± 0%      37.0 ± 0%     ~     (all equal)
AggregationWithOTTL/sum_cumulative-10                227 ± 0%       263 ± 0%  +15.66%  (p=0.008 n=5+5)
AggregationWithOTTL/sum_delta-10                     229 ± 0%       264 ± 0%  +15.28%  (p=0.016 n=4+5)
AggregationWithOTTL/histogram_cumulative-10          244 ± 1%       281 ± 2%  +15.33%  (p=0.008 n=5+5)
AggregationWithOTTL/histogram_delta-10               246 ± 1%       279 ± 2%  +13.07%  (p=0.008 n=5+5)
AggregationWithOTTL/exphistogram_cumulative-10       256 ± 0%       292 ± 0%  +14.06%  (p=0.029 n=4+4)
AggregationWithOTTL/exphistogram_delta-10            265 ± 0%       301 ± 0%  +13.58%  (p=0.016 n=4+5)
AggregationWithOTTL/summary_enabled-10               240 ± 0%       275 ± 0%  +14.58%  (p=0.008 n=5+5)
AggregationWithOTTL/summary_passthrough-10          37.0 ± 0%      37.0 ± 0%     ~     (all equal)

TL;DR, there is still ~20% performance degradation with overflows. We could eliminate the overflow path when it is not defined but that will only give us a false sense of improvement since, for our use case, we would always have overflows defined.

I have another PR open here which optimizes the pebble options for our use case and that will have improvements to this PR too but the diff between with and without overflow will still be approximately same.

[axw]

Getting back up to speed after PTO - mostly minor comments

[axw]

Should this also be AppendBinary? Then we could avoid a slice allocation & copy in Value.Marshal

[lahsivjar]

Sounds good. We won't be able to fully utilize this though since it is not possible to estimate the size of the tracker prior to marshaling, so we could still end up reallocating the slice due to axiomhq/hyperloglog#44

[lahsivjar]

I tried this out, and the resulting code to encode trackers gets a bit messy because there is no way to calculate marshaled size beforehand. Maybe we could defer this to when we have the AppendBinary in the hll (the PR you created) and a way to estimate the size?

[axw]

Yeah no worries, we can follow up on this

[lahsivjar]

@axw I was already planning to add resource datapoints limits but now I am questioning myself if scope limits and scope datapoint limits are required or even useful, maybe just having resource limits and resource datapoints limits are enough for all use-cases? Do you see a valid case where scope limits would be useful?

[axw]

@axw I was already planning to add resource datapoints limits but now I am questioning myself if scope limits and scope datapoint limits are required or even useful, maybe just having resource limits and resource datapoints limits are enough for all use-cases? Do you see a valid case where scope limits would be useful?

I don't think that we would encounter high-cardinality scopes, but it may still be useful to have an explicit limit for scopes independent of metrics/datapoints, to avoid high cardinality metric names or dimensions from causing scopes to be washed out. That's the same reason why I think we should have a metric limit.

So what I have in mind for limits is to following the hierarchy: resource, resource-scope, resource-scope-metric, resource-scope-metric-datapoint.

Why were you planning to add a resource-datapoint limit?

[axw]

Looks good now, thanks for the updates. Let's resolve the discussion about limits and then get it in :)

[lahsivjar]

So what I have in mind for limits is to following the hierarchy: resource, resource-scope, resource-scope-metric, resource-scope-metric-datapoint.

This also sounds good to me.

Why were you planning to add a resource-datapoint limit?

The main intention for adding this was to provide a way to cater to cases when we are not sure how many scope metrics can be within a scope but we don't want to waste the limit either i.e. if we expect number of scope metrics to have a high variance we could just do resource datapoint limit and prevent losing the unutilized limits for resources with lower number of scopes... but now that I think about this, I am not sure if scope metrics are meant to be high cardinality - maybe I got ahead of the problem without thinking it is actually a problem 😓 .

[axw]

So what I have in mind for limits is to following the hierarchy: resource, resource-scope, resource-scope-metric, resource-scope-metric-datapoint.

This also sounds good to me.

Do you want to do that in this PR, or in a followup?

[lahsivjar]

Do you want to do that in this PR, or in a followup?

I will do it in a followup