[processor/lsminterval] Optimize pebble usage

[lahsivjar]

Detailed description on the changes available in godocs

[lahsivjar]

Bench stats with main:

name                                            old time/op    new time/op    delta
Aggregation/sum_cumulative-10                     10.0µs ± 5%     8.9µs ± 3%  -10.39%  (p=0.008 n=5+5)
Aggregation/sum_delta-10                          10.0µs ± 3%     8.9µs ± 2%  -10.93%  (p=0.008 n=5+5)
Aggregation/histogram_cumulative-10               10.1µs ± 3%    10.1µs ± 2%     ~     (p=0.421 n=5+5)
Aggregation/histogram_delta-10                    10.4µs ± 2%    10.5µs ± 9%     ~     (p=1.000 n=5+5)
Aggregation/exphistogram_cumulative-10            11.1µs ± 2%    10.1µs ± 5%   -8.58%  (p=0.008 n=5+5)
Aggregation/exphistogram_delta-10                 11.2µs ± 2%    10.2µs ± 3%   -8.91%  (p=0.008 n=5+5)
Aggregation/summary_enabled-10                    10.3µs ± 1%     9.7µs ± 4%   -5.82%  (p=0.008 n=5+5)
Aggregation/summary_passthrough-10                1.30µs ± 2%    1.32µs ± 7%     ~     (p=0.127 n=5+5)
AggregationWithOTTL/sum_cumulative-10             12.0µs ± 3%    10.9µs ± 1%   -9.78%  (p=0.008 n=5+5)
AggregationWithOTTL/sum_delta-10                  12.2µs ± 4%    10.7µs ± 2%  -12.46%  (p=0.008 n=5+5)
AggregationWithOTTL/histogram_cumulative-10       12.5µs ± 0%    11.3µs ± 1%   -9.97%  (p=0.016 n=4+5)
AggregationWithOTTL/histogram_delta-10            11.8µs ± 7%    11.5µs ± 4%     ~     (p=0.690 n=5+5)
AggregationWithOTTL/exphistogram_cumulative-10    13.2µs ± 6%    11.7µs ± 3%  -11.21%  (p=0.008 n=5+5)
AggregationWithOTTL/exphistogram_delta-10         13.2µs ± 4%    11.6µs ± 1%  -11.94%  (p=0.008 n=5+5)
AggregationWithOTTL/summary_enabled-10            12.7µs ± 5%    11.2µs ± 3%  -11.36%  (p=0.008 n=5+5)
AggregationWithOTTL/summary_passthrough-10        1.40µs ± 5%    1.30µs ± 4%   -7.28%  (p=0.016 n=5+5)

name                                            old alloc/op   new alloc/op   delta
Aggregation/sum_cumulative-10                     17.6kB ± 3%    10.6kB ± 0%  -40.00%  (p=0.008 n=5+5)
Aggregation/sum_delta-10                          17.6kB ± 1%    10.6kB ± 0%  -39.94%  (p=0.008 n=5+5)
Aggregation/histogram_cumulative-10               19.4kB ± 2%    15.8kB ± 2%  -18.64%  (p=0.008 n=5+5)
Aggregation/histogram_delta-10                    19.4kB ± 1%    15.8kB ± 2%  -18.60%  (p=0.008 n=5+5)
Aggregation/exphistogram_cumulative-10            21.7kB ± 0%    12.9kB ± 0%  -40.32%  (p=0.008 n=5+5)
Aggregation/exphistogram_delta-10                 22.2kB ± 1%    13.3kB ± 0%  -39.88%  (p=0.008 n=5+5)
Aggregation/summary_enabled-10                    18.5kB ± 1%    12.0kB ±13%  -34.98%  (p=0.016 n=4+5)
Aggregation/summary_passthrough-10                1.38kB ± 0%    1.36kB ± 1%     ~     (p=0.079 n=5+5)
AggregationWithOTTL/sum_cumulative-10             20.2kB ± 1%    13.0kB ± 0%  -35.70%  (p=0.008 n=5+5)
AggregationWithOTTL/sum_delta-10                  20.2kB ± 1%    12.9kB ± 1%  -35.98%  (p=0.008 n=5+5)
AggregationWithOTTL/histogram_cumulative-10       22.1kB ± 3%    15.0kB ± 0%  -31.98%  (p=0.008 n=5+5)
AggregationWithOTTL/histogram_delta-10            21.9kB ± 2%    15.0kB ± 0%  -31.48%  (p=0.008 n=5+5)
AggregationWithOTTL/exphistogram_cumulative-10    23.0kB ± 5%    15.2kB ± 0%  -33.65%  (p=0.016 n=5+4)
AggregationWithOTTL/exphistogram_delta-10         23.6kB ± 3%    14.7kB ± 0%  -37.68%  (p=0.008 n=5+5)
AggregationWithOTTL/summary_enabled-10            20.7kB ± 4%    13.4kB ± 0%  -35.31%  (p=0.008 n=5+5)
AggregationWithOTTL/summary_passthrough-10        1.38kB ± 1%    1.37kB ± 1%     ~     (p=0.056 n=5+5)

name                                            old allocs/op  new allocs/op  delta
Aggregation/sum_cumulative-10                        199 ± 1%       124 ± 0%  -37.63%  (p=0.008 n=5+5)
Aggregation/sum_delta-10                             200 ± 0%       125 ± 0%  -37.62%  (p=0.008 n=5+5)
Aggregation/histogram_cumulative-10                  211 ± 1%       174 ± 1%  -17.52%  (p=0.008 n=5+5)
Aggregation/histogram_delta-10                       213 ± 0%       176 ± 1%  -17.56%  (p=0.008 n=5+5)
Aggregation/exphistogram_cumulative-10               228 ± 0%       142 ± 0%  -37.72%  (p=0.008 n=5+5)
Aggregation/exphistogram_delta-10                    237 ± 0%       151 ± 0%  -36.29%  (p=0.008 n=5+5)
Aggregation/summary_enabled-10                       212 ± 0%       146 ±14%  -30.94%  (p=0.016 n=4+5)
Aggregation/summary_passthrough-10                  37.0 ± 0%      35.4 ± 2%   -4.32%  (p=0.008 n=5+5)
AggregationWithOTTL/sum_cumulative-10                227 ± 0%       151 ± 0%  -33.60%  (p=0.008 n=5+5)
AggregationWithOTTL/sum_delta-10                     229 ± 0%       152 ± 0%     ~     (p=0.079 n=4+5)
AggregationWithOTTL/histogram_cumulative-10          244 ± 1%       165 ± 0%  -32.38%  (p=0.008 n=5+5)
AggregationWithOTTL/histogram_delta-10               246 ± 1%       166 ± 0%  -32.63%  (p=0.008 n=5+5)
AggregationWithOTTL/exphistogram_cumulative-10       256 ± 0%       169 ± 0%     ~     (p=0.079 n=4+5)
AggregationWithOTTL/exphistogram_delta-10            265 ± 0%       178 ± 0%     ~     (p=0.079 n=4+5)
AggregationWithOTTL/summary_enabled-10               240 ± 0%       160 ± 0%  -33.33%  (p=0.008 n=5+5)
AggregationWithOTTL/summary_passthrough-10          37.0 ± 0%      35.6 ± 2%   -3.78%  (p=0.008 n=5+5)

[lahsivjar]

[For reviewers] I have gone a bit high here, this would mean we would require 64*2=128MB of memory at-least (due to max number of memtables as configured via pebbleMemTableStopWritesThreshold) for memtable. This should be good for most of our use cases - also, we could expose these configs in the component configuration for tweaking in future but I haven't done that in this PR.