[Security Solution] Detections fail when an index is missing a primary shard

[BenB196]

Describe the bug:

If there is an index that is used in a detection rule, and that index is missing a primary shard (cluster in red state), the detection rules that use that index will fail with the error:

Bulk Indexing of signals failed: index: "<index_with_primary_shard_missing>" reason: "[<node_with_missing_shard>][127.0.0.1:9300][indices:data/read/search[phase/query]]" type: "no_shard_available_action_exception" name: "<detection_rule_name>" id: "ac117e42-a91f-11eb-a252-7f35a8822039" rule id: "5e552599-ddec-4e14-bad1-28aa42404388" signals index: ".siem-signals-security"

This results in the detection rule(s) not running, and therefore not detecting things.

Kibana/Elasticsearch Stack version:

Elasticsearch: 7.13.1
Kibana: 7.13.1

Server OS version:

OS: Docker/ECK

Browser and Browser OS versions:

N/A

Elastic Endpoint version:

N/A

Original install method (e.g. download page, yum, from source, etc.):

ECK

Functional Area (e.g. Endpoint management, timelines, resolver, etc.):

Detections

Steps to reproduce:

1. Create a detection rule that uses an index pattern. ex: filebeat-*
2. Create two indices, filebeat-1, filebeat-2
3. Start detection rule
4. Cause a primary shard to fail on filebeat-1
5. Detection rule will fail

Current behavior:

Detection rule fails when a primary shard is missing. This results in no detections occurring at all.

Expected behavior:

Detection rule should still work, but should instead show as warning. Detections should still trigger from the available indices.

Screenshots (if relevant):

N/A

Errors in browser console (if relevant):

N/A

Provide logs and/or server output (if relevant):

N/A

Any additional context (logs, chat logs, magical formulas, etc.):

I know that a red cluster state is bad, but sometimes it happens, and I would expect a security solution to best effort maintain monitoring of the environment while the red state is resolved.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[yctercero]

@BenB196 I know this is far too late, but could you please elaborate what did you observe when ignore_unavailable was used? We've recently had some similar questions come up and are working with ES to figure out how to best handle these cases as the rules themselves shouldn't be concerned with the underlying details of the backing indices.