[Security Solution][Detections] Pre-packaged Threshold Rules schema out of date resulting in errors on Rule Management page

[spong]

With #92667, the Threshold Rule schema has changed, so we'll need to coordinate with @elastic/protections on updating the pre-packaged threshold rules. Until the rules are updated, this issue (#93325) where the find API returns an error on the Rules Management page will continue to occur once the pre-packaged rules have been loaded.

Kibana Version: 7.12 BC2

Old Schema

export const threatsOrUndefined = t.union([threats, t.undefined]);
export type ThreatsOrUndefined = t.TypeOf<typeof threatsOrUndefined>;

export const threshold = t.intersection([
  t.exact(
    t.type({
      field: t.union([t.string, t.array(t.string)]),
      value: PositiveIntegerGreaterThanZero,
    })
  ),
  t.exact(
    t.partial({
      cardinality_field: t.union([t.string, t.array(t.string), t.undefined, t.null]),
      cardinality_value: t.union([PositiveInteger, t.undefined, t.null]), // TODO: cardinality_value should be set if cardinality_field is set
    })
  ),
]);
// TODO: codec to transform threshold field string to string[] ?
export type Threshold = t.TypeOf<typeof threshold>;

export const thresholdOrUndefined = t.union([threshold, t.undefined]);
export type ThresholdOrUndefined = t.TypeOf<typeof thresholdOrUndefined>;

New Schema

export const threatsOrUndefined = t.union([threats, t.undefined]);
export type ThreatsOrUndefined = t.TypeOf<typeof threatsOrUndefined>;

export const thresholdField = t.exact(
  t.type({
    field: t.union([t.string, t.array(t.string)]), // Covers pre- and post-7.12
    value: PositiveIntegerGreaterThanZero,
  })
);
export type ThresholdField = t.TypeOf<typeof thresholdField>;

export const thresholdFieldNormalized = t.exact(
  t.type({
    field: t.array(t.string),
    value: PositiveIntegerGreaterThanZero,
  })
);
export type ThresholdFieldNormalized = t.TypeOf<typeof thresholdFieldNormalized>;

export const thresholdCardinalityField = t.exact(
  t.type({
    field: t.string,
    value: PositiveInteger,
  })
);
export type ThresholdCardinalityField = t.TypeOf<typeof thresholdCardinalityField>;

export const threshold = t.intersection([
  thresholdField,
  t.exact(
    t.partial({
      cardinality: t.union([t.array(thresholdCardinalityField), t.null]),
    })
  ),
]);
export type Threshold = t.TypeOf<typeof threshold>;

export const thresholdOrUndefined = t.union([threshold, t.undefined]);
export type ThresholdOrUndefined = t.TypeOf<typeof thresholdOrUndefined>;

export const thresholdNormalized = t.intersection([
  thresholdFieldNormalized,
  t.exact(
    t.partial({
      cardinality: t.union([t.array(thresholdCardinalityField), t.null]),
    })
  ),
]);
export type ThresholdNormalized = t.TypeOf<typeof thresholdNormalized>;

export const thresholdNormalizedOrUndefined = t.union([thresholdNormalized, t.undefined]);
export type ThresholdNormalizedOrUndefined = t.TypeOf<typeof thresholdNormalizedOrUndefined>;

Sample of new threshold rule

{
  "author": [],
  "created_at": "2021-03-03T01:14:51.146Z",
  "updated_at": "2021-03-03T02:04:02.428Z",
  "created_by": "howdy@elastic.co",
  "description": "desc",
  "enabled": true,
  "false_positives": [],
  "from": "now-900s",
  "id": "da284640-7bbd-11eb-be1b-fd715f2b5a92",
  "immutable": false,
  "interval": "5m",
  "rule_id": "a65e4d3e-5e7c-4167-83e6-ae380a8d92ee",
  "output_index": ".siem-signals-siem-estc-dev-default",
  "max_signals": 100,
  "risk_score": 73,
  "risk_score_mapping": [],
  "name": "3-Way Cluster",
  "references": [],
  "severity": "high",
  "severity_mapping": [],
  "updated_by": "howdy@elastic.co",
  "tags": [
    "clusters"
  ],
  "to": "now",
  "type": "threshold",
  "threat": [],
  "version": 3,
  "exceptions_list": [
    {
      "id": "b854be20-7bc4-11eb-be1b-fd715f2b5a92",
      "list_id": "294a0a3b-524e-40db-ba8f-47bb92b40e24",
      "type": "detection",
      "namespace_type": "single"
    }
  ],
  "actions": [],
  "filters": [],
  "index": [
    ".siem-signals*"
  ],
  "license": "",
  "meta": {
    "from": "10m",
    "kibana_siem_app_url": "https://kibana.siem.estc.dev/app/security"
  },
  "throttle": "no_actions",
  "status": "succeeded",
  "status_date": "2021-03-03T02:12:01.069Z",
  "last_success_at": "2021-03-03T02:12:01.069Z",
  "last_success_message": "succeeded",
  "query": "signal.rule.type: query",
  "language": "kuery",
  "threshold": {
    "field": [
      "host.name"
    ],
    "value": 2,
    "cardinality": [
      {
        "field": "signal.rule.name",
        "value": 2
      }
    ]
  }
}

cc @MadameSheema @madirey @peluja1012

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[spong]

This has been confirmed to be a non-issue, closing -- apologies for the noise here.

[ghost]

Bug Conversion:

Test case is not required for this ticket as this is a non-issue.

Thanks!!