store: Added benchmarks for .Series and expanded matchers. #2070

bwplotka · 2020-01-28T17:42:56Z

Changes

Added ExpandedPostingsMatchers benchmarks. This function and only this
will be affected by index-header changes.
Added store Series test. This can be later expanded for tests for different StoreAPIs not only Store GW.
Added few interfaces to improve testability to Store GW bucket struct.
Fixed time filtering edge case. Now request mint=0, maxt=0 should succeed.

ExpandedPostings

Script for running those:

#!/usr/bin/env bash

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

cd ${DIR}/../

RUN="${1}-exppostings"
mkdir -p ${DIR}/bench_outs/${RUN}

for TEST in BenchmarkBucketIndexReader_ExpandedPostings/binary_header BenchmarkBucketIndexReader_ExpandedPostings/index_json
do
echo "Running ${TEST} run ${RUN}"
go test -bench=${TEST} -run=^$ -benchmem -memprofile ${DIR}/bench_outs/${RUN}/memprofile${TEST//\//-}.out -timeout 2h -benchtime 60s ./pkg/store | tee  ${DIR}/bench_outs/${RUN}/bench${TEST//\//-}.out
done

sed 's/BenchmarkBucketIndexReader_ExpandedPostings\/binary_header/BenchmarkBucketIndexReader_ExpandedPostings/g' ${DIR}/bench_outs/${RUN}/benchBenchmarkBucketIndexReader_ExpandedPostings-binary_header.out > ${DIR}/bench_outs/${RUN}/bench_new.out
sed 's/BenchmarkBucketIndexReader_ExpandedPostings\/index_json/BenchmarkBucketIndexReader_ExpandedPostings/g' ${DIR}/bench_outs/${RUN}/benchBenchmarkBucketIndexReader_ExpandedPostings-index_json.out > ${DIR}/bench_outs/${RUN}/bench_old.out

benchcmp ${DIR}/bench_outs/${RUN}/bench_old.out ${DIR}/bench_outs/${RUN}/bench_new.out | tee ${DIR}/bench_outs/${RUN}/bench.out

Results:

benchmark                                                                         old ns/op     new ns/op     delta
BenchmarkBucketIndexReader_ExpandedPostings/n="1"-12                              3842599       4396053       +14.40%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",j="foo"-12                      32690723      31569625      -3.43%
BenchmarkBucketIndexReader_ExpandedPostings/j="foo",n="1"-12                      32505999      33343759      +2.58%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",j!="foo"-12                     200367800     229696038     +14.64%
BenchmarkBucketIndexReader_ExpandedPostings/i=~".*"-12                            153160181     188532296     +23.09%
BenchmarkBucketIndexReader_ExpandedPostings/i=~".+"-12                            380726946     463532800     +21.75%
BenchmarkBucketIndexReader_ExpandedPostings/i=~""-12                              421439451     512133098     +21.52%
BenchmarkBucketIndexReader_ExpandedPostings/i!=""-12                              308473739     407513630     +32.11%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".*",j="foo"-12              136999349     163355090     +19.24%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".*",i!="2",j="foo"-12       161095504     178444900     +10.77%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i!=""-12                        123643237     154754514     +25.16%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i!="",j="foo"-12                141086986     175977956     +24.73%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",j="foo"-12              212058205     244589372     +15.34%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~"1.+",j="foo"-12             33981603      46508363      +36.86%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",i!="2",j="foo"-12       268266004     323914937     +20.74%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",i!~"2.*",j="foo"-12     306779915     347659338     +13.33%

benchmark                                                                         old allocs     new allocs     delta
BenchmarkBucketIndexReader_ExpandedPostings/n="1"-12                              73             75             +2.74%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",j="foo"-12                      108            112            +3.70%
BenchmarkBucketIndexReader_ExpandedPostings/j="foo",n="1"-12                      108            112            +3.70%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",j!="foo"-12                     142            148            +4.23%
BenchmarkBucketIndexReader_ExpandedPostings/i=~".*"-12                            96             98             +2.08%
BenchmarkBucketIndexReader_ExpandedPostings/i=~".+"-12                            100159         300159         +199.68%
BenchmarkBucketIndexReader_ExpandedPostings/i=~""-12                              100114         300115         +199.77%
BenchmarkBucketIndexReader_ExpandedPostings/i!=""-12                              100156         300156         +199.69%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".*",j="foo"-12              149            155            +4.03%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".*",i!="2",j="foo"-12       181            193            +6.63%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i!=""-12                        100174         300176         +199.65%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i!="",j="foo"-12                100204         300208         +199.60%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",j="foo"-12              100207         300211         +199.59%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~"1.+",j="foo"-12             11285          33511          +196.95%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",i!="2",j="foo"-12       100240         300247         +199.53%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",i!~"2.*",j="foo"-12     111349         333577         +199.58%

benchmark                                                                         old bytes     new bytes     delta
BenchmarkBucketIndexReader_ExpandedPostings/n="1"-12                              11379240      11379278      +0.00%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",j="foo"-12                      23527225      23527336      +0.00%
BenchmarkBucketIndexReader_ExpandedPostings/j="foo",n="1"-12                      23527224      23527339      +0.00%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",j!="foo"-12                     90635328      90636435      +0.00%
BenchmarkBucketIndexReader_ExpandedPostings/i=~".*"-12                            285362375     334956762     +17.38%
BenchmarkBucketIndexReader_ExpandedPostings/i=~".+"-12                            331578558     384372836     +15.92%
BenchmarkBucketIndexReader_ExpandedPostings/i=~""-12                              182041232     234834884     +29.00%
BenchmarkBucketIndexReader_ExpandedPostings/i!=""-12                              331576878     384371194     +15.92%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".*",j="foo"-12              92242749      141837796     +53.77%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".*",i!="2",j="foo"-12       94340164      143955415     +52.59%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i!=""-12                        126308942     179103370     +41.80%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i!="",j="foo"-12                138456800     191251255     +38.13%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",j="foo"-12              138458444     191253140     +38.13%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~"1.+",j="foo"-12             33852975      83803000      +147.55%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",i!="2",j="foo"-12       207664951     260478496     +25.43%
BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",i!~"2.*",j="foo"-12     212691177     315435644     +48.31%

Series

Script:

#!/usr/bin/env bash

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

cd ${DIR}/../

RUN="${1}-series"
mkdir -p ${DIR}/bench_outs/${RUN}

for TEST in BenchmarkSeries/binary_header BenchmarkSeries/index_json
do
echo "Running ${TEST} run ${RUN}"
go test -bench=${TEST} -run=^$ -benchmem -memprofile ${DIR}/bench_outs/${RUN}/memprofile${TEST//\//-}.out -timeout 2h -benchtime 30s ./pkg/store | tee  ${DIR}/bench_outs/${RUN}/bench${TEST//\//-}.out
done

sed 's/BenchmarkSeries\/binary_header/BenchmarkSeries/g' ${DIR}/bench_outs/${RUN}/benchBenchmarkSeries-binary_header.out > ${DIR}/bench_outs/${RUN}/bench_new.out
sed 's/BenchmarkSeries\/index_json/BenchmarkSeries/g' ${DIR}/bench_outs/${RUN}/benchBenchmarkSeries-index_json.out > ${DIR}/bench_outs/${RUN}/bench_old.out

benchcmp ${DIR}/bench_outs/${RUN}/bench_old.out ${DIR}/bench_outs/${RUN}/bench_new.out | tee ${DIR}/bench_outs/${RUN}/bench.out

Results:

Running BenchmarkSeries/binary_header run 0-series
goos: linux
goarch: amd64
pkg: github.com/thanos-io/thanos/pkg/store
BenchmarkSeries/binary_header/1of10000000-12         	      10	3405980704 ns/op	1577961956 B/op	20076939 allocs/op
BenchmarkSeries/binary_header/10of10000000-12        	      10	3338306252 ns/op	1577902473 B/op	20076885 allocs/op
BenchmarkSeries/binary_header/100of10000000-12       	      10	3381093365 ns/op	1577926803 B/op	20077301 allocs/op
BenchmarkSeries/binary_header/1000of10000000-12      	      10	3346731304 ns/op	1578388380 B/op	20081755 allocs/op
BenchmarkSeries/binary_header/10000of10000000-12     	       9	3387770880 ns/op	1587132363 B/op	20127273 allocs/op
BenchmarkSeries/binary_header/100000of10000000-12    	      10	3509686626 ns/op	1671451827 B/op	20580914 allocs/op
BenchmarkSeries/binary_header/1000000of10000000-12   	       8	4379223614 ns/op	2514264246 B/op	25115266 allocs/op
BenchmarkSeries/binary_header/10000000of10000000-12  	       3	12498205945 ns/op	15882149677 B/op	130614810 allocs/op
PASS
ok  	github.com/thanos-io/thanos/pkg/store	481.055s
Running BenchmarkSeries/index_json run 0-series
goos: linux
goarch: amd64
pkg: github.com/thanos-io/thanos/pkg/store
BenchmarkSeries/index_json/1of10000000-12         	      13	2519404503 ns/op	1437956498 B/op	15076920 allocs/op
BenchmarkSeries/index_json/10of10000000-12        	      14	2570713484 ns/op	1437918778 B/op	15076972 allocs/op
BenchmarkSeries/index_json/100of10000000-12       	      14	2462235340 ns/op	1437948738 B/op	15077374 allocs/op
BenchmarkSeries/index_json/1000of10000000-12      	      15	2409482348 ns/op	1438429929 B/op	15081921 allocs/op
BenchmarkSeries/index_json/10000of10000000-12     	      15	2447382076 ns/op	1447084683 B/op	15127167 allocs/op
BenchmarkSeries/index_json/100000of10000000-12    	      14	2452156730 ns/op	1530657506 B/op	15580889 allocs/op
BenchmarkSeries/index_json/1000000of10000000-12   	      10	3234136594 ns/op	2379163824 B/op	20115223 allocs/op
BenchmarkSeries/index_json/10000000of10000000-12  	signal: killed
FAIL	github.com/thanos-io/thanos/pkg/store	569.329s
FAIL
benchmark                                old ns/op      new ns/op      delta
BenchmarkSeries/1of10000000-12           2519404503     3405980704     +35.19%
BenchmarkSeries/10of10000000-12          2570713484     3338306252     +29.86%
BenchmarkSeries/100of10000000-12         2462235340     3381093365     +37.32%
BenchmarkSeries/1000of10000000-12        2409482348     3346731304     +38.90%
BenchmarkSeries/10000of10000000-12       2447382076     3387770880     +38.42%
BenchmarkSeries/100000of10000000-12      2452156730     3509686626     +43.13%
BenchmarkSeries/1000000of10000000-12     3234136594     4379223614     +35.41%

benchmark                                old allocs     new allocs     delta
BenchmarkSeries/1of10000000-12           15076920       20076939       +33.16%
BenchmarkSeries/10of10000000-12          15076972       20076885       +33.16%
BenchmarkSeries/100of10000000-12         15077374       20077301       +33.16%
BenchmarkSeries/1000of10000000-12        15081921       20081755       +33.15%
BenchmarkSeries/10000of10000000-12       15127167       20127273       +33.05%
BenchmarkSeries/100000of10000000-12      15580889       20580914       +32.09%
BenchmarkSeries/1000000of10000000-12     20115223       25115266       +24.86%

benchmark                                old bytes      new bytes      delta
BenchmarkSeries/1of10000000-12           1437956498     1577961956     +9.74%
BenchmarkSeries/10of10000000-12          1437918778     1577902473     +9.74%
BenchmarkSeries/100of10000000-12         1437948738     1577926803     +9.73%
BenchmarkSeries/1000of10000000-12        1438429929     1578388380     +9.73%
BenchmarkSeries/10000of10000000-12       1447084683     1587132363     +9.68%
BenchmarkSeries/100000of10000000-12      1530657506     1671451827     +9.20%
BenchmarkSeries/1000000of10000000-12     2379163824     2514264246     +5.68%

Signed-off-by: Bartlomiej Plotka bwplotka@gmail.com

pkg/store/bucket_test.go

bwplotka · 2020-01-29T16:15:35Z

You can ignore CI error, it's because I need to rebase on #2077 once merged.

brian-brazil · 2020-01-29T16:43:29Z

I don't see anything obviously wrong.

bwplotka · 2020-01-30T13:15:06Z

First iteration of .Series result is fun! (:

bash _dev/bench_series.sh 0
Running BenchmarkSeries/binary_header run 0-series
goos: linux
goarch: amd64
pkg: github.com/thanos-io/thanos/pkg/store
BenchmarkSeries_MillionsSeriesWithOneSample/binary_header/1of10000000-12         	      10	3170961965 ns/op	1577961656 B/op	20076938 allocs/op
BenchmarkSeries_MillionsSeriesWithOneSample/binary_header/10of10000000-12        	      10	3233990066 ns/op	1577881177 B/op	20076844 allocs/op
BenchmarkSeries_MillionsSeriesWithOneSample/binary_header/100of10000000-12       	      10	3179363645 ns/op	1577940600 B/op	20077338 allocs/op
BenchmarkSeries_MillionsSeriesWithOneSample/binary_header/1000of10000000-12      	      10	3118202426 ns/op	1578423198 B/op	20081876 allocs/op
BenchmarkSeries_MillionsSeriesWithOneSample/binary_header/10000of10000000-12     	      10	3139396100 ns/op	1587140179 B/op	20127240 allocs/op
BenchmarkSeries_MillionsSeriesWithOneSample/binary_header/100000of10000000-12    	      10	3253384227 ns/op	1671065876 B/op	20580979 allocs/op
BenchmarkSeries_MillionsSeriesWithOneSample/binary_header/1000000of10000000-12   	       8	4105534012 ns/op	2518368762 B/op	25115331 allocs/op
bucket_test.go:1223: unexpected error: rpc error: code = Aborted desc = fetch series for block 01DZV3V8TSFV4008VD6D6YVNKP: preload chunks: read range for 0: allocate chunk bytes: pool exhausted

--- FAIL: BenchmarkSeries_MillionsSeriesWithOneSample/binary_header/10000000of10000000
--- FAIL: BenchmarkSeries_MillionsSeriesWithOneSample/binary_header
--- FAIL: BenchmarkSeries_MillionsSeriesWithOneSample
FAIL
exit status 1
FAIL	github.com/thanos-io/thanos/pkg/store	390.390s
FAIL
Running BenchmarkSeries/index_json run 0-series
goos: linux
goarch: amd64
pkg: github.com/thanos-io/thanos/pkg/store
BenchmarkSeries_MillionsSeriesWithOneSample/index_json/1of10000000-12         	      15	2356337341 ns/op	1437954038 B/op	15076911 allocs/op
BenchmarkSeries_MillionsSeriesWithOneSample/index_json/10of10000000-12        	      15	2371328532 ns/op	1437874976 B/op	15076814 allocs/op
BenchmarkSeries_MillionsSeriesWithOneSample/index_json/100of10000000-12       	      15	2387921632 ns/op	1437963037 B/op	15077418 allocs/op
BenchmarkSeries_MillionsSeriesWithOneSample/index_json/1000of10000000-12      	      15	2362966154 ns/op	1438438411 B/op	15081966 allocs/op
BenchmarkSeries_MillionsSeriesWithOneSample/index_json/10000of10000000-12     	      15	2395617626 ns/op	1447102662 B/op	15127229 allocs/op
BenchmarkSeries_MillionsSeriesWithOneSample/index_json/100000of10000000-12    	      15	2404596601 ns/op	1530925085 B/op	15580972 allocs/op
BenchmarkSeries_MillionsSeriesWithOneSample/index_json/1000000of10000000-12   	      12	3237704906 ns/op	2374237739 B/op	20115173 allocs/op
bucket_test.go:1223: unexpected error: rpc error: code = Aborted desc = fetch series for block 01DZV49S7ZXV2YF4VS4NQ1DSZS: preload chunks: read range for 0: allocate chunk bytes: pool exhausted

2GB of memory to fetch 1mln series and that is WITH the chunk bytes pool (: Trying without...

alloc_space:

bwplotka · 2020-01-30T14:27:01Z

Without pooling bytes for chunks it' madness:

Running BenchmarkSeries/binary_header run 1-series
goos: linux
goarch: amd64
pkg: github.com/thanos-io/thanos/pkg/store
BenchmarkSeries/binary_header/1of10000000-12         	      10	3284197505 ns/op	1577936804 B/op	20076854 allocs/op
BenchmarkSeries/binary_header/10of10000000-12        	      10	3295759539 ns/op	1577945545 B/op	20076895 allocs/op
BenchmarkSeries/binary_header/100of10000000-12       	      10	3302932189 ns/op	1578028665 B/op	20077475 allocs/op
BenchmarkSeries/binary_header/1000of10000000-12      	      10	3275058090 ns/op	1578476056 B/op	20081734 allocs/op
BenchmarkSeries/binary_header/10000of10000000-12     	      10	3254601925 ns/op	1587646488 B/op	20127175 allocs/op
BenchmarkSeries/binary_header/100000of10000000-12    	      10	3366360233 ns/op	1676253709 B/op	20580883 allocs/op
BenchmarkSeries/binary_header/1000000of10000000-12   	       8	4232465798 ns/op	2566112362 B/op	25115298 allocs/op
BenchmarkSeries/binary_header/10000000of10000000-12  	       3	13466568311 ns/op	15881951480 B/op	130613948 allocs/op
PASS
ok  	github.com/thanos-io/thanos/pkg/store	480.663s
Running BenchmarkSeries/index_json run 1-series
goos: linux
goarch: amd64
pkg: github.com/thanos-io/thanos/pkg/store
BenchmarkSeries/index_json/1of10000000-12         	      15	2424057657 ns/op	1437937921 B/op	15076856 allocs/op
BenchmarkSeries/index_json/10of10000000-12        	      13	2538203730 ns/op	1437956158 B/op	15076931 allocs/op
BenchmarkSeries/index_json/100of10000000-12       	      13	2447718936 ns/op	1437993884 B/op	15077353 allocs/op
BenchmarkSeries/index_json/1000of10000000-12      	      15	2507760852 ns/op	1438558841 B/op	15082019 allocs/op
BenchmarkSeries/index_json/10000of10000000-12     	      13	2488399436 ns/op	1447666612 B/op	15127243 allocs/op
BenchmarkSeries/index_json/100000of10000000-12    	      14	2664945098 ns/op	1536270808 B/op	15580945 allocs/op
BenchmarkSeries/index_json/1000000of10000000-12   	      10	3407783584 ns/op	2426086508 B/op	20115231 allocs/op
BenchmarkSeries/index_json/10000000of10000000-12  	signal: killed
FAIL	github.com/thanos-io/thanos/pkg/store	511.756s
FAIL

It looks good: index-cache did not complete due to OOM (: vs the index-header manage to do it 3 times for 30s ❤️

bwplotka · 2020-01-30T14:36:41Z

Flame graph for tests with chunk bytes pool (so default for Thanos):

Interesting questions / major mem usages:

read bytes from object storage takes a lot of space: I think we need to reuse more bytes / pool - this will help a lot 👍
It would be nice to cache / pool symbols at some point... They take significant amount as well.
we are sorting.. I guess labels here:

sort.Slice(s.lset, func(i, j int) bool {
			return s.lset[i].Name < s.lset[j].Name
})

This is becasue we are adding external labels. It takes as well some mem, but really not good idea how to avoid this - I think labels has to be sorted overall.

bwplotka · 2020-01-30T14:44:23Z

OK, this is ready go. In separate PR I will remove index_json cases. We want to get rid of this, so no point spending time in benchmarking this.

Also in separate PR I will improve things. This is our Baseline.

PTAL @GiedriusS @brancz @squat @metalmatze @kakkoyun

brancz

Nice start. If we can it would be great if in the future we would be able to benchmark not only operations but also bytes per second etc., which involves writing a bit more involved benchmarks but I think this is a good baseline.

roidelapluie · 2020-01-30T15:32:18Z

website/data/adopters.yml

@@ -78,3 +78,9 @@ adopters:
 - name: Banzai Cloud
  url: https://banzaicloud.com/
  logo: banzaicloud.png
+- name: Jurumani Solutions


unrelated to this PR??

Sorry chained PRs problems -> Moved this PR to target master now since the base PR was merged. (: Now should be fine! Sorry.

Please PTAL

bwplotka · 2020-01-30T18:15:27Z

pkg/store/bucket.go

 	start := mint
 	for _, b := range s.blocks[i] {
 		if b.meta.MaxTime <= mint {
 			continue
 		}
-		if b.meta.MinTime >= maxt {
+		// NOTE: Block intervals are half-open: [b.MinTime, b.MaxTime).
+		if b.meta.MinTime > maxt {


@GiedriusS @brancz @squat

I think this was tiny bug we had. Request for Series with time selection e.g [mint=t1, maxt=t1] was matching no block. Now if there is any that starts with t1 it will be matched to return potentially single sample.

Do you mean [mint=t1,maxt=t1]?

Sorry, yes: Do you think that's fair?

## Changes * Added ExpandedPostingsMatchers benchmarks. This function and only this will be affected by index-header changes. * Added store Series test. This can be later expanded for tests for different StoreAPIs not only Store GW. * Added few interfaces to improve testability to Store GW bucket struct. * Fixed time filtering edge case. Now request mint=0, maxt=0 should succeed. ## ExpandedPostings Script for running those: ``` #!/usr/bin/env bash DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" cd ${DIR}/../ RUN="${1}-exppostings" mkdir -p ${DIR}/bench_outs/${RUN} for TEST in BenchmarkBucketIndexReader_ExpandedPostings/binary_header BenchmarkBucketIndexReader_ExpandedPostings/index_json do echo "Running ${TEST} run ${RUN}" go test -bench=${TEST} -run=^$ -benchmem -memprofile ${DIR}/bench_outs/${RUN}/memprofile${TEST//\//-}.out -timeout 2h -benchtime 60s ./pkg/store | tee ${DIR}/bench_outs/${RUN}/bench${TEST//\//-}.out done sed 's/BenchmarkBucketIndexReader_ExpandedPostings\/binary_header/BenchmarkBucketIndexReader_ExpandedPostings/g' ${DIR}/bench_outs/${RUN}/benchBenchmarkBucketIndexReader_ExpandedPostings-binary_header.out > ${DIR}/bench_outs/${RUN}/bench_new.out sed 's/BenchmarkBucketIndexReader_ExpandedPostings\/index_json/BenchmarkBucketIndexReader_ExpandedPostings/g' ${DIR}/bench_outs/${RUN}/benchBenchmarkBucketIndexReader_ExpandedPostings-index_json.out > ${DIR}/bench_outs/${RUN}/bench_old.out benchcmp ${DIR}/bench_outs/${RUN}/bench_old.out ${DIR}/bench_outs/${RUN}/bench_new.out | tee ${DIR}/bench_outs/${RUN}/bench.out ``` Results: ``` benchmark old ns/op new ns/op delta BenchmarkBucketIndexReader_ExpandedPostings/n="1"-12 3842599 4396053 +14.40% BenchmarkBucketIndexReader_ExpandedPostings/n="1",j="foo"-12 32690723 31569625 -3.43% BenchmarkBucketIndexReader_ExpandedPostings/j="foo",n="1"-12 32505999 33343759 +2.58% BenchmarkBucketIndexReader_ExpandedPostings/n="1",j!="foo"-12 200367800 229696038 +14.64% BenchmarkBucketIndexReader_ExpandedPostings/i=~".*"-12 153160181 188532296 +23.09% BenchmarkBucketIndexReader_ExpandedPostings/i=~".+"-12 380726946 463532800 +21.75% BenchmarkBucketIndexReader_ExpandedPostings/i=~""-12 421439451 512133098 +21.52% BenchmarkBucketIndexReader_ExpandedPostings/i!=""-12 308473739 407513630 +32.11% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".*",j="foo"-12 136999349 163355090 +19.24% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".*",i!="2",j="foo"-12 161095504 178444900 +10.77% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i!=""-12 123643237 154754514 +25.16% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i!="",j="foo"-12 141086986 175977956 +24.73% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",j="foo"-12 212058205 244589372 +15.34% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~"1.+",j="foo"-12 33981603 46508363 +36.86% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",i!="2",j="foo"-12 268266004 323914937 +20.74% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",i!~"2.*",j="foo"-12 306779915 347659338 +13.33% benchmark old allocs new allocs delta BenchmarkBucketIndexReader_ExpandedPostings/n="1"-12 73 75 +2.74% BenchmarkBucketIndexReader_ExpandedPostings/n="1",j="foo"-12 108 112 +3.70% BenchmarkBucketIndexReader_ExpandedPostings/j="foo",n="1"-12 108 112 +3.70% BenchmarkBucketIndexReader_ExpandedPostings/n="1",j!="foo"-12 142 148 +4.23% BenchmarkBucketIndexReader_ExpandedPostings/i=~".*"-12 96 98 +2.08% BenchmarkBucketIndexReader_ExpandedPostings/i=~".+"-12 100159 300159 +199.68% BenchmarkBucketIndexReader_ExpandedPostings/i=~""-12 100114 300115 +199.77% BenchmarkBucketIndexReader_ExpandedPostings/i!=""-12 100156 300156 +199.69% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".*",j="foo"-12 149 155 +4.03% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".*",i!="2",j="foo"-12 181 193 +6.63% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i!=""-12 100174 300176 +199.65% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i!="",j="foo"-12 100204 300208 +199.60% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",j="foo"-12 100207 300211 +199.59% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~"1.+",j="foo"-12 11285 33511 +196.95% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",i!="2",j="foo"-12 100240 300247 +199.53% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",i!~"2.*",j="foo"-12 111349 333577 +199.58% benchmark old bytes new bytes delta BenchmarkBucketIndexReader_ExpandedPostings/n="1"-12 11379240 11379278 +0.00% BenchmarkBucketIndexReader_ExpandedPostings/n="1",j="foo"-12 23527225 23527336 +0.00% BenchmarkBucketIndexReader_ExpandedPostings/j="foo",n="1"-12 23527224 23527339 +0.00% BenchmarkBucketIndexReader_ExpandedPostings/n="1",j!="foo"-12 90635328 90636435 +0.00% BenchmarkBucketIndexReader_ExpandedPostings/i=~".*"-12 285362375 334956762 +17.38% BenchmarkBucketIndexReader_ExpandedPostings/i=~".+"-12 331578558 384372836 +15.92% BenchmarkBucketIndexReader_ExpandedPostings/i=~""-12 182041232 234834884 +29.00% BenchmarkBucketIndexReader_ExpandedPostings/i!=""-12 331576878 384371194 +15.92% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".*",j="foo"-12 92242749 141837796 +53.77% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".*",i!="2",j="foo"-12 94340164 143955415 +52.59% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i!=""-12 126308942 179103370 +41.80% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i!="",j="foo"-12 138456800 191251255 +38.13% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",j="foo"-12 138458444 191253140 +38.13% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~"1.+",j="foo"-12 33852975 83803000 +147.55% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",i!="2",j="foo"-12 207664951 260478496 +25.43% BenchmarkBucketIndexReader_ExpandedPostings/n="1",i=~".+",i!~"2.*",j="foo"-12 212691177 315435644 +48.31% ``` ## Series Script: ``` #!/usr/bin/env bash DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" cd ${DIR}/../ RUN="${1}-series" mkdir -p ${DIR}/bench_outs/${RUN} for TEST in BenchmarkSeries/binary_header BenchmarkSeries/index_json do echo "Running ${TEST} run ${RUN}" go test -bench=${TEST} -run=^$ -benchmem -memprofile ${DIR}/bench_outs/${RUN}/memprofile${TEST//\//-}.out -timeout 2h -benchtime 30s ./pkg/store | tee ${DIR}/bench_outs/${RUN}/bench${TEST//\//-}.out done sed 's/BenchmarkSeries\/binary_header/BenchmarkSeries/g' ${DIR}/bench_outs/${RUN}/benchBenchmarkSeries-binary_header.out > ${DIR}/bench_outs/${RUN}/bench_new.out sed 's/BenchmarkSeries\/index_json/BenchmarkSeries/g' ${DIR}/bench_outs/${RUN}/benchBenchmarkSeries-index_json.out > ${DIR}/bench_outs/${RUN}/bench_old.out benchcmp ${DIR}/bench_outs/${RUN}/bench_old.out ${DIR}/bench_outs/${RUN}/bench_new.out | tee ${DIR}/bench_outs/${RUN}/bench.out ``` Results: ``` Running BenchmarkSeries/binary_header run 0-series goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/store BenchmarkSeries/binary_header/1of10000000-12 10 3405980704 ns/op 1577961956 B/op 20076939 allocs/op BenchmarkSeries/binary_header/10of10000000-12 10 3338306252 ns/op 1577902473 B/op 20076885 allocs/op BenchmarkSeries/binary_header/100of10000000-12 10 3381093365 ns/op 1577926803 B/op 20077301 allocs/op BenchmarkSeries/binary_header/1000of10000000-12 10 3346731304 ns/op 1578388380 B/op 20081755 allocs/op BenchmarkSeries/binary_header/10000of10000000-12 9 3387770880 ns/op 1587132363 B/op 20127273 allocs/op BenchmarkSeries/binary_header/100000of10000000-12 10 3509686626 ns/op 1671451827 B/op 20580914 allocs/op BenchmarkSeries/binary_header/1000000of10000000-12 8 4379223614 ns/op 2514264246 B/op 25115266 allocs/op BenchmarkSeries/binary_header/10000000of10000000-12 3 12498205945 ns/op 15882149677 B/op 130614810 allocs/op PASS ok github.com/thanos-io/thanos/pkg/store 481.055s Running BenchmarkSeries/index_json run 0-series goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/store BenchmarkSeries/index_json/1of10000000-12 13 2519404503 ns/op 1437956498 B/op 15076920 allocs/op BenchmarkSeries/index_json/10of10000000-12 14 2570713484 ns/op 1437918778 B/op 15076972 allocs/op BenchmarkSeries/index_json/100of10000000-12 14 2462235340 ns/op 1437948738 B/op 15077374 allocs/op BenchmarkSeries/index_json/1000of10000000-12 15 2409482348 ns/op 1438429929 B/op 15081921 allocs/op BenchmarkSeries/index_json/10000of10000000-12 15 2447382076 ns/op 1447084683 B/op 15127167 allocs/op BenchmarkSeries/index_json/100000of10000000-12 14 2452156730 ns/op 1530657506 B/op 15580889 allocs/op BenchmarkSeries/index_json/1000000of10000000-12 10 3234136594 ns/op 2379163824 B/op 20115223 allocs/op BenchmarkSeries/index_json/10000000of10000000-12 signal: killed FAIL github.com/thanos-io/thanos/pkg/store 569.329s FAIL benchmark old ns/op new ns/op delta BenchmarkSeries/1of10000000-12 2519404503 3405980704 +35.19% BenchmarkSeries/10of10000000-12 2570713484 3338306252 +29.86% BenchmarkSeries/100of10000000-12 2462235340 3381093365 +37.32% BenchmarkSeries/1000of10000000-12 2409482348 3346731304 +38.90% BenchmarkSeries/10000of10000000-12 2447382076 3387770880 +38.42% BenchmarkSeries/100000of10000000-12 2452156730 3509686626 +43.13% BenchmarkSeries/1000000of10000000-12 3234136594 4379223614 +35.41% benchmark old allocs new allocs delta BenchmarkSeries/1of10000000-12 15076920 20076939 +33.16% BenchmarkSeries/10of10000000-12 15076972 20076885 +33.16% BenchmarkSeries/100of10000000-12 15077374 20077301 +33.16% BenchmarkSeries/1000of10000000-12 15081921 20081755 +33.15% BenchmarkSeries/10000of10000000-12 15127167 20127273 +33.05% BenchmarkSeries/100000of10000000-12 15580889 20580914 +32.09% BenchmarkSeries/1000000of10000000-12 20115223 25115266 +24.86% benchmark old bytes new bytes delta BenchmarkSeries/1of10000000-12 1437956498 1577961956 +9.74% BenchmarkSeries/10of10000000-12 1437918778 1577902473 +9.74% BenchmarkSeries/100of10000000-12 1437948738 1577926803 +9.73% BenchmarkSeries/1000of10000000-12 1438429929 1578388380 +9.73% BenchmarkSeries/10000of10000000-12 1447084683 1587132363 +9.68% BenchmarkSeries/100000of10000000-12 1530657506 1671451827 +9.20% BenchmarkSeries/1000000of10000000-12 2379163824 2514264246 +5.68% ``` Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

bwplotka requested review from GiedriusS, povilasv, brancz and squat January 29, 2020 13:03

roidelapluie reviewed Jan 29, 2020

View reviewed changes

pkg/store/bucket_test.go Outdated Show resolved Hide resolved

bwplotka force-pushed the index-hdr branch from 3b6b4fd to 54a650f Compare January 29, 2020 16:17

bwplotka changed the base branch from master to testutils January 29, 2020 16:18

bwplotka force-pushed the index-hdr branch from 54a650f to 056f3e7 Compare January 30, 2020 09:01

bwplotka force-pushed the index-hdr branch from 056f3e7 to 53fa260 Compare January 30, 2020 13:47

bwplotka force-pushed the index-hdr branch from 53fa260 to 6d79de4 Compare January 30, 2020 14:42

bwplotka marked this pull request as ready for review January 30, 2020 14:43

bwplotka force-pushed the testutils branch from da4090d to b442f45 Compare January 30, 2020 14:53

brancz approved these changes Jan 30, 2020

View reviewed changes

bwplotka force-pushed the testutils branch from b442f45 to e232db1 Compare January 30, 2020 15:03

bwplotka force-pushed the index-hdr branch from 6d79de4 to fcb41f0 Compare January 30, 2020 15:30

roidelapluie reviewed Jan 30, 2020

View reviewed changes

bwplotka force-pushed the index-hdr branch from fcb41f0 to 3701046 Compare January 30, 2020 15:34

bwplotka changed the base branch from testutils to master January 30, 2020 15:35

bwplotka force-pushed the index-hdr branch 3 times, most recently from 878dde2 to eb76770 Compare January 30, 2020 18:12

bwplotka commented Jan 30, 2020

View reviewed changes

bwplotka force-pushed the index-hdr branch 2 times, most recently from f56978c to 22dcfe9 Compare February 3, 2020 13:55

bwplotka force-pushed the index-hdr branch from 22dcfe9 to 60eed61 Compare February 3, 2020 15:40

bwplotka force-pushed the index-hdr branch from 60eed61 to 5b7991a Compare February 3, 2020 16:27

bwplotka merged commit 56a1fb6 into master Feb 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

store: Added benchmarks for .Series and expanded matchers. #2070

store: Added benchmarks for .Series and expanded matchers. #2070

bwplotka commented Jan 28, 2020 •

edited

Loading

bwplotka commented Jan 29, 2020 •

edited

Loading

brian-brazil commented Jan 29, 2020

bwplotka commented Jan 30, 2020 •

edited

Loading

bwplotka commented Jan 30, 2020

bwplotka commented Jan 30, 2020

bwplotka commented Jan 30, 2020 •

edited

Loading

brancz left a comment

roidelapluie Jan 30, 2020

bwplotka Jan 30, 2020

bwplotka Jan 30, 2020 •

edited

Loading

GiedriusS Jan 31, 2020

bwplotka Feb 3, 2020

store: Added benchmarks for .Series and expanded matchers. #2070

store: Added benchmarks for .Series and expanded matchers. #2070

Conversation

bwplotka commented Jan 28, 2020 • edited Loading

Changes

ExpandedPostings

Series

bwplotka commented Jan 29, 2020 • edited Loading

brian-brazil commented Jan 29, 2020

bwplotka commented Jan 30, 2020 • edited Loading

bwplotka commented Jan 30, 2020

bwplotka commented Jan 30, 2020

bwplotka commented Jan 30, 2020 • edited Loading

brancz left a comment

Choose a reason for hiding this comment

roidelapluie Jan 30, 2020

Choose a reason for hiding this comment

bwplotka Jan 30, 2020

Choose a reason for hiding this comment

bwplotka Jan 30, 2020 • edited Loading

Choose a reason for hiding this comment

GiedriusS Jan 31, 2020

Choose a reason for hiding this comment

bwplotka Feb 3, 2020

Choose a reason for hiding this comment

bwplotka commented Jan 28, 2020 •

edited

Loading

bwplotka commented Jan 29, 2020 •

edited

Loading

bwplotka commented Jan 30, 2020 •

edited

Loading

bwplotka commented Jan 30, 2020 •

edited

Loading

bwplotka Jan 30, 2020 •

edited

Loading