[alerting] migrating an alert with a large alert param into 7.13.0 will cause a migration error

[pmuellr]

Kibana version: 7.13.0

Elasticsearch version: 7.13.0

Describe the bug:

In 7.13.0, we changed the type of the alert saved object params property, from type: object; enabled: false to type: flattened in PR #92036.

kibana/x-pack/plugins/alerting/server/saved_objects/mappings.json

Lines 49 to 51 in a0ddca8

	"params": {
	"type": "flattened"
	},

Unfortunately, this will cause a problem for any field in the params which is greater than 32766 byte long; we just saw such an alert fail in a migration in an internal repo.

FATAL  Error: Unable to complete saved object migrations for the [.kibana] index. 
Error: pickupUpdatedMappings task failed with the following failures:
[
  {
    "index": ".kibana_7.13.0_001",
    "type": "_doc",
    "id": "alert:1b5a5b52-5fdb-4a17-9ecb-909083978c60",
    "cause": {
      "type": "illegal_argument_exception",
      "reason": "Document contains at least one immense term in field=\"alert.params\" (whose UTF8 encoding is longer
than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The
prefix of the first immense term is: '[bytes elited]...', original message: bytes can be at most 32766 in length;
got 92326",
      "caused_by": {
        "type": "max_bytes_length_exceeded_exception",
        "reason": "max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 92326"
      }
    },
    "status": 400
  }
]

I haven't repro'd it yet, but I would expect that you could cause this to fail with a normal alert creation with a param over the length limit. If that doesn't fail, or fails differently, I think we'd want to try to repro the migration failure directly, by creating an alert with a huge param in 7.12, then migrating to 7.13.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[pmuellr]

We're hoping we can "fix" this by adding an ignore_above to the field definition ... https://www.elastic.co/guide/en/elasticsearch/reference/current/flattened.html#flattened-params

[pmuellr]

Was able to repro the error by creating an alert with a 100K param. Used the es query alert - when creating a new one, it populates it's query parameter with a simple query, so I inserted 100K space characters before the final } (in case we did a trim() on the value).

{
  "query":{
    "match_all" : {}
  }
}

So, assuming that if we can prevent this from happening with this repro during create, it will also fix the migration issue.

[pmuellr]

Added "ignore_above": 4096 just after the type parameter here, and was able to save the alert, see it run, re-edit it again - so seems like it's going to work.

kibana/x-pack/plugins/alerting/server/saved_objects/mappings.json

Lines 49 to 51 in 58f45ee

	"params": {
	"type": "flattened"
	},

I wasn't sure what value to use for ignore_above - most uses in Kibana use 1024, but they're for relatively "simple" fields, and I think we want to account for more. The max will be in the 32K range, but didn't want to really go that high. 4K seemed pretty reasonable, 8K would be fine for me as well.

This should only impact the searching of alerts via the params, nothing else operationally.

Now to write a test ...

[mikecote]

I wasn't sure what value to use for ignore_above - most uses in Kibana use 1024, but they're for relatively "simple" fields, and I think we want to account for more. The max will be in the 32K range, but didn't want to really go that high. 4K seemed pretty reasonable, 8K would be fine for me as well.

From my understanding, this ignore_above tells Elasticsearch to skip analyzing values that are above X? If that's the case, whatever number we put would be the threshold before exact matches stop working. 1k, 4k or 8k seem pretty big for an exact match query?