[Response Ops][Alerting] Should framework alerts as data write out data flattened keys

[ymao1]

Currently, the rule registry writes out documents with flattened keys like:

{
  "kibana.alert.rule.name": "test"
}

The alerting framework writes out documents as an object

{
  "kibana": {
    "alert": {
      "rule": {
        "name": "test"
      }
    }
  }
}

When considering this, we looked at what might happen if we had mixed formats in the same index, where doc1 was flattened and doc2 was unflattened and it seemed to make no difference when reading. Reading and writing sources as objects also allow us to apply typescript typings to the ES search requests.

However, when performing an update by query on these documents, we could potentially get into a mixed state within the same document. For example, updating an object source with flattened keys:

{
  "kibana.alert.status": "untracked",
  "kibana": {
    "alert": {
      "status": "active"
    }
  }
}

We would have to account for this scenario in the painless scripts we use to update alerts, which is possible but is it necessary? Currently we have just the stack rules onboarded onto the framework alerts-as-data so the only manual update that would be performed on them would be in this new PR #164788. Should we switch to writing flattened alerts before onboarding more rule types?

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[ymao1]

Currently, AAD docs are updated in a few places:

• kibana.alert.case_ids - this field is not populated when an alert doc is first created, so updating this field may result in a doc with mixed flatted & unflattened keys (in the case of an ES query alert visible in the Observability alerts table) but no conflicting information within the doc
• kibana.alert.workflow_status - this field is updated in two places
  • as part of a cases update - this update by query checks whether ctx._source['${ALERT_WORKFLOW_STATUS}'] is null so it will not update alert documents that are unflattened; this means we will not have conflicting information within a doc but ES query AAD docs will not have its workflow status correctly updated when its associated case is updated.
  • as part of security solutions open/close signals route - this only updates detection rules AAD, which are currently all flattened keys so no risk of conflicting information within a doc
• kibana.alert.workflow_tags - this is updated as part of security solutions set alert tags route which only updates detection rule AAD docs, which are currently all flattened keys so no risk of conflicting information within a doc
• kibana.alert.status - this is included in this PR and checks for flattened vs unflattened keys so no risk of conflicting information within the doc

[ymao1]

Going to split this into 2 PRs:

1. Change the framework alerts client to write and expect flattened alert data - [Response Ops][Alerting] Update framework alerts client to write flattened alerts docs #167439
2. Followup to handle the edge case/bug we'll have for the ES query rule type where there may be unflattened docs written in 8.10 that need to be updated (for ongoing active alerts and recovered alerts). These will currently not show up anywhere in the UI (only 8.11 created ES query rules with metrics or logs consumer will show up in O11y alert table) and there is no stack alerts table yet to display the 8.10 created alerts.