[RAC] Existence of index template should be verified before index is created

[dgieselaar]

When writing alerts, the rule registry tries to create a concrete index for a write target if it doesn't exist (after it has caught a index_not_found_exception from bulk). It currently does not check whether an index template exists before it does so - which means that it might create an index without any mappings, leading to partial failures when this index matches a target that e.g. sorts on @timestamp. To prevent this, we should check right before creating the index whether a valid index template exists. We can probably do so by calling the _index_template/_simulate_index API:

const { body: simulateResponse } = await clusterClient.transport.request({
  method: 'POST',
  path: `/_index_template/_simulate_index/${concreteIndexName}`,
});

if (isEmpty(simulateResponse)) {
  throw new Error(
    'Index template simulation resulted in empty object, possibly due to missing index template.'
  );
}

Alternatively, we can check for the existence of technical fields, but a non-empty mappings object is probably good enough for now.

cc @tsg @jasonrhodes @spong @smith

[tsg]

FYI also @banderror

[jasonrhodes]

@tsg @spong is this something someone on the Security side can fix alongside already planned improvements to the registry / rule data tools, or do you want me to put someone on this? Either way works, just want to make sure it's covered.

[spong]

@banderror should be able to address this as part of the follow-up convergence once #98353 is merged, so we should be good. If this ends up falling out of scope of that effort for some reason we can identify someone to ensure it still makes it 👍

[banderror]

Yes, I can try to address this as part of the follow-up convergence. Might need some help from someone who's more familiar with Elasticsearch though, but we'll figure this out. Thanks for mentioning me here 👍

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[banderror]

Hey everyone, FYI ownership of this ticket and other tickets related to rule_registry (like #101016) now goes to the Detection Alerts area (Team:Detection Alerts label). Please ping @peluja1012 and @marshallmain if you have any questions.