[Alerting] Expose SO _version to clients

[madirey]

Currently, users of the security_solution plugin can run into a number of race conditions when multiple users attempt to update a rule at the same time. There's no way for us to circumvent these with OCC, as the _version field is not exposed for the alerts via the Alerting Client.

Here's one area where we could run into trouble:

kibana/x-pack/plugins/security_solution/server/lib/detection_engine/rules/update_rules.ts

Line 101 in 52bbfff

const update = await alertsClient.update({

The solution would be to expose the _version in the get and find APIs, and allow that to be passed in when updating. Then we can handle 409 conflict scenarios by refreshing and having the user try the rule update again.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[mikecote]

Note: alerting UIs should also take advantage of such implementation.

[mikecote]

Relates to #50260

[pmuellr]

There was an indirect reference to this issue from a PR today - #77838 (comment)

I happened to think we haven't even proposed what we might want to do here. Given some context on what the referenced PR is "fixing", here are some questions:

• do we support OCC in the all mutating APIs (ie, add an optional version option that maps to SO OCC); mutating APIs include update(), enable(), muteAll(), etc, and delete() (ES supports OCC for delete anyway)? Or do we have a subset of those APIs that support OCC - eg, update() supports it, none of the other methods support it?

• what does it mean when an API that takes optional OCC but it's not passed on the call - do we not use OCC in the SO calls at all, and so just do overwrites? Or do we use it internally like we do today within each API (eg, enable() does a GET at the beginning and uses the version when doing it's final UPDATE). In the latter case, do we retry a few times (like the referenced PR does), or return the conflict exception immediately?

My take is that I think we just want OCC for the update() call, and when it's not passed, we retry like the referenced PR. When the OCC is passed, then we rethrow the conflict error immediately. The other mutating methods would work like the referenced PR - retry a few times.

My rationale is that update() is the only method that updates user-controlled data - the other mutating methods (except delete()) are updating framework information - and all of this is stored in the same SO. OCC is super important for user-controlled data, to ensure users making simultaneous don't overwrite themselves. Less clear for me that it's useful for framework-y calls - would someone really use OCC with enable()? Even if we supported that, this is going to be painful for callers who will be getting this because of the upcoming background updates to the alert execution status (to be stored in the alert SO). There will be a lot more opportunities for conflicts than just conflicts from other Kibana users.

[mikecote]

Moving from 7.12 - Candidates to 7.x - Candidates.

[mikecote]

Moving from 7.x - Candidates to 8.x - Candidates (Backlog) after the latest 7.x planning session.

[gmmorris]

Worth discussing this issue further given this comment by @mikecote .

It sounds like there's a concern that exposing the version will lead to bad UX because the rule could easily be updated in the background by the system (when executing it) and that will likely clash with user's updating other parts of the rule SO.

TBH It feels like this is our internal implementation leaking, and I think it's worth taking a closer look at the impact of the underlying implementation (updating the status on the SO there by changing the version) and see if this approach makes sense longer term... but obviously changing this now is very costly.