[Alerting] Investigate risk of performance regressions from share-capable saved object types

[mikecote]

See #113743

We should investigate the performance impact on our usage of the SO APIs (running rules, CRUD rules, etc). We should also work with the @elastic/security-detections-response team and see if they experience performance regressions when doing bulk operations using our rules client (ex: enable all).

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[chrisronline]

I did some investigation and synced with @mikecote on the approach here.

Rather than dive into each individual call and understand the exact performance implications from the start, we decided to look holistically at performance before and after these changes.

As a way to start, we want to measure performance of operations that might happen quite frequently and small increases in underlying function calls could make a significant difference.

To test the changes, I used master as is which reflects using the code with potential latency and then I tested it against master but I changed all namespaceType: 'multiple-isolated' to namespaceType: 'single' (thanks to the tip from @joeportner) and ran the same scenario (Note: I started fresh with each creation scenario where I spun up a brand new ES)

I looked at three main areas:

1. Mass creation of connectors and rules
2. Mass update of connectors and rules
3. TM health stats with mass amount of rules/connectors running

For my testing, I used 200 rules and connectors (one unique connector for each unique rule)

namespaceType	Creation time	Update time	p50 drift
multiple-isolated	643s	420s	62169s
single	637s	422s	63093s

As a result of this testing, I'm concluding that there isn't a significant performance change with the new share-capable related changes.