[Alerting + Task Manager] Benchmarking 7.14

[gmmorris]

Following the release of 7.14 and with it the fresh perf work we've done we decided we need to do the following:

1. Run fresh and comprehensive performance tests on the 7.14 release
2. Use the results of the perf test as a basis for a sizing your Kibana cluster for alerting blogpost, similar to https://www.elastic.co/blog/benchmarking-and-sizing-your-elasticsearch-cluster-for-logs-and-metrics

note: 2021-08-09: we split item 2 off to a separate issue: [alerting] public facing doc on alerting performance #107979

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[gmmorris]

Following the latest RAC leads sync (held 6th April), I think we should bump this to a later point so we can include Alerts-as-Data in the benchmarks.

This does raise some questions about how we want to measure our API performance.
Should we add timing measurements for the various APIs that rely on Alerts-as-data?
Such as the APIs fetching alerts for the Alert table?

[gmmorris]

It looks like ES have been able to address the issue we identified in UpdateByQuery: elastic/elasticsearch#63671

We expect this to address the degradation we've identified when there are more than 30 Kibana instances or so, so it's something we should validate at part of this issue.

[gmmorris]

Bumped to 7.14 😬

[pmuellr]

The stress tester needed some updates, as ecctl has changed a bit since the last time we ran the tester, and a few enhancements have been added:

commits:

• pmuellr/kbn-alert-load@b098672
• pmuellr/kbn-alert-load@d7b92d4

changes:

• added 7.14 as the top-level version
• changed running alerts at 1m interval, to 3s interval; similar
  change to decrease the number of rules we actually create;
  thinking is, we can simulate more alerts by running them at a
  smaller interval
• increased wait for deployments to finish creation from 5 minutes to 10
• changed how kibana config is set (change to ecctl data shape)
• now finds closest match to platform's choice of RAM usage

[pmuellr]

Here are three runs I made with the tester; I didn't see any significant regressions, nor any significant increase in performance - as expected

• 210809-223758-tm-max-workers-400 - kbn-alert-load report.pdf
  compare running with 10, 15, 20 task manager max_workers

• 210805-154404-deployment-size-1000 - kbn-alert-load report.pdf
  compare running with 4, 6, 8, 10 Kibanas, each running with 8GB RAM

• 210805-135403-stack-versions-500 - kbn-alert-load report.pdf
  compare running 7.14, 7.13, 7.12, 7.11