Merge pull request #1015 from grafana/benchmarks-cluster-queries

Benchmarks cluster queries

Author: Dieterbe

benchmarks/cluster-queries/README.md

@@ -0,0 +1,36 @@
+## cluster-queries benchmark
+
+## goal
+
+realistic query workload to see
+where we allocate (and keep in use) significant amounts of memory
+
+in particular for:
+1) optimizing pointSlicePool
+2) optimizing chunkcache
+
+## running the benchmark
+
+```
+./launch.sh docker-cluster
+```
+
+wait for stack to be up.
+
+```
+./fakemetrics backfill --kafka-mdm-addr localhost:9092 --kafka-mdm-v2=false --offset $((2*366*24))h --period 1800s --speedup 360000 --mpo 500
+```
+
+this should complete in about 3 minutes.
+load MT dashboard and confirm it ingested data at 200kHz for 3 minutes
+
+when done: in grafana look at `some.id.of.a.metric.{1,72,199,320,438};id=*;some=tag` to confirm data looks good (over 2y)
+
+finally: `./run.sh`
+
+## observations:
+
+* memory usage grows during test but does not release afterwards (!).
+  in line with cache, but much more than cache reports (~50MB). profile shows ~50~60% of allocations are in chunk cache
+
+

benchmarks/cluster-queries/alerting-input

@@ -0,0 +1,3 @@
+GET http://localhost:6061/render?target=some.id.of.a.metric.1*;id=*;some=tag&from=-30s
+GET http://localhost:6061/render?target=some.id.of.a.metric.22*;id=*;some=tag&from=-60s
+GET http://localhost:6061/render?target=some.id.of.a.metric.33*;id=*;some=tag&from=-1h

benchmarks/cluster-queries/run-alerting.sh

@@ -0,0 +1 @@
+cat alerting-input | vegeta attack -rate 20 -duration 30s > alerting-out

benchmarks/cluster-queries/run-exploration.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+
+
+function gen_input() {
+	# we have 500 series and want to select 10 out of them
+	for i in {1..300};
+	do
+		patt="$(($RANDOM % 5))$(($RANDOM % 10))*"
+		echo "GET http://localhost:6061/render?target=some.id.of.a.metric.$patt;some=tag&from=-2y"
+	done
+}
+
+input=$(gen_input)
+echo "$input" | vegeta attack -rate 1 -duration 30s > exploration-out

benchmarks/cluster-queries/run.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+# alerting use case: frequent reads of the same, recent data
+./run-alerting.sh &
+# exploration case: infrequent, random data, longer timeframe
+./run-exploration.sh
+
+sleep 3
+echo "alerting:"
+cat alerting-out | vegeta report
+echo "exploration:"
+cat exploration-out | vegeta report