[Security Solution] [Detections] Combine multiple timestamp searches into single request

[dhurley14]

Summary

Ref: #93550

We've been sending two separate queries when a timestamp override is provided to a rule definition in order to ensure we are also querying data that might have been sent over but is missing that timestamp override field for some reason.

The proposed change in this PR merges the logic of these two queries into one single search.

Added a few integration tests which should cover all cases but please take a look at those and point out any missing cases that should be covered.

Test data

POST myfakeindex-1/_doc
{
  "message": "hello world 1"
}

POST myfakeindex-2/_doc
{
  "message": "hello world 2",
  "event": {
    "ingested": "2021-04-13T21:17:56.235Z"
  }
}

POST myfakeindex-3/_doc
{
  "message": "hello world 3",
  "@timestamp": "2021-04-13T21:17:56.235Z"
}

POST myfakeindex-4/_doc
{
  "message": "hello world 2",
  "@timestamp": "2021-04-13T21:17:56.235Z",
  "event": {
    "ingested": "2021-04-13T21:17:56.235Z"
  }
}

Rule Query

GET myfa*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "must": [],
            "filter": [
              {
                "match_all": {}
              }
            ],
            "should": [],
            "must_not": []
          }
        },
        {
          "bool": {
            "filter": [
              {
                "bool": {
                  "should": [
                    {
                      "range": {
                        "event.ingested": {
                          "lte": "2021-04-15T13:26:40.428Z",
                          "gte": "2021-04-15T12:26:30.428Z",
                          "format": "strict_date_optional_time"
                        }
                      }
                    },
                    {
                      "bool": {
                        "filter": [
                          {
                            "range": {
                              "@timestamp": {
                                "lte": "2021-04-15T13:26:40.428Z",
                                "gte": "2021-04-15T12:26:30.428Z",
                                "format": "strict_date_optional_time"
                              }
                            }
                          },
                          {
                            "bool": {
                              "must_not": {
                                "exists": {
                                  "field": "event.ingested"
                                }
                              }
                            }
                          }
                        ]
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              }
            ]
          }
        },
        {
          "match_all": {}
        }
      ]
    }
  },
  "fields": [
    {
      "field": "*",
      "include_unmapped": true
    }
  ],
  "sort": [
    {
      "event.ingested": {
        "order": "asc",
        "unmapped_type": "date"
      }
    },
    {
      "@timestamp": {
        "order": "asc",
        "unmapped_type": "date"
      }
    }
  ]
}

After posting the above documents and executing the query in the search profiler in dev tools we can see that the query is behaving as we expect.

Screenshot

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[dhurley14]

typscript complains that the format field is not applicable to the @timestamp field. I think the type is wrong.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[dhurley14]

@elasticmachine merge upstream

[marshallmain]

LGTM! Good integration test coverage 👍

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: d155fed
• Storybooks Preview

Metrics [docs]

✅ unchanged

History

• 💔 Build #120996 failed 40b9b27
• 💔 Build #120966 failed b3d932a
• 💔 Build #120829 failed d29c16f
• 💛 Build #120537 was flaky df74b28651dbc072a803c9c07b85f4c5d6f21c4f
• 💚 Build #120018 succeeded 40ce84983b518bf07102f0ee3cb598406407a23b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @dhurley14

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.