[Security Solution] Truncated error messages during rule execution

[xcrzx]

Summary

During the search phase of rule execution, thrown error messages are truncated, making it difficult to identify the root cause of the error.

See SDHs for example:

1. https://github.com/elastic/sdh-security-team/issues/508
2. https://github.com/elastic/sdh-security-team/issues/501
3. https://github.com/elastic/sdh-security-team/issues/500

All of them have error messages logged without root cause in both the Kibana logs and the rule execution log:

search_after_bulk_create threw an error ResponseError: search_phase_execution_exception: [siem.queryRule][AWS Config Resource Deletion][rule id 92e3cf40-75bf-11ed-89b7-0bebc4b98e27][rule uuid 7024e2a0-315d-4334-bb1a-552d604f27bc][exec id 456fdc0b-91e6-4261-9815-5a88a27288f7][space default]

We need to find a way to extract useful information from Elasticsearch errors and log it with the error. Sometimes it is specified in the reason field, for example:

{
  "message": "status_exception",
  "statusCode": 400,
  "attributes": {
    "type": "status_exception",
    "reason": "error while executing search",
    "caused_by": {
      "type": "search_phase_execution_exception",
      "reason": "Partial shards failure",
      "phase": "query",
      "grouped": true,
      "failed_shards": [
        {
          "shard": 0,
          "index": ".ds-elastic-cloud-logs-8-2022.03.05-000001",
          "node": "FAfhiKFgQRub0foBuI0llg",
          "reason": {
            "type": "illegal_argument_exception",
            "reason": "Field [timestamp] of type [keyword] is not supported for aggregation [date_histogram]"
          }
        }
      ],
      "caused_by": {
        "type": "",
        "reason": ": Field [timestamp] of type [keyword] is not supported for aggregation [date_histogram]",
        "caused_by": {
          "type": "illegal_argument_exception",
          "reason": "Field [timestamp] of type [keyword] is not supported for aggregation [date_histogram]"
        }
      }
    }
  }
}

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[banderror]

All these issues could have been fixed by elastic/elastic-transport-js#52 which was propagated to Kibana in #148521. We need to verify that it fixes the issues mentioned in this ticket.

[banderror]

@marshallmain and @elastic/security-detections-response-alerts folks, would you be able to help here with verifying that the error cases from the linked SDHs are now resolved? I think this is more related to rule execution logic and how errors are created, rather than the rule execution logging mechanism and features, so I'd rely on your knowledge of the detection engine here.