[Security Solution] Rule schemas tech debt

[banderror]

Related to: #80792

Summary

TL;DR: We have multiple rule schemas on various levels which increases the costs of maintaining existing rule types and adding new rule types or parameters. We should have just one schema for reading rules + a few modifications of it for creating rules, importing rules, installing prebuilt rules, and such.

Problem

On a high level, we have the following rule schemas:

1. Schema of the data we store to Elasticsearch via RulesClient. These are the rule types and parameters that are known to the Alerting Framework.
2. Schema of rules that we return from our HTTP endpoints.
3. Schema of rules that can be accepted by the import endpoint.
4. Schema of rules that can be accepted by the "add prepackaged rules" endpoint.
5. POST, PUT and PATCH schemas of rules that can be accepted by the CRUD endpoints.
6. A completely separate schema of rules on the FE side.
7. Schemas of forms used on the Rule Creation/Editing pages.

Every time we need to add a new rule parameter (doesn't matter a common one or a rule type-specific one) or implement a new rule type, we need to correctly adjust the above 7 schemas + a big number of related files. The problem is:

• It requires a lot of effort: it's about 40-80 files to change. Example PRs:
  • [Security Solution][Detections] Add new fields to the rule model: Related Integrations, Required Fields, and Setup #132409
  • [Security Solution] Expose EQL search API configs in EQL rules #130339 #132247
  • [Security Solution][Alerts] New terms security rule type #134526
• There's a high chance of human error. It's easy to overlook something because not all the functions are statically typed.

Solution

We should reduce the 7 schemas we have right now to the following ones:

1. We should have only one single schema for reading rules from the API. It should live in the common folder. This would be our "source of truth" and the main rule schema. We shouldn't have a separate FE schema.
2. We will have a few modifications of it for creating rules, importing rules, installing prebuilt rules, and such. These should be derivatives from the main schema.
3. We will have a schema for the data we store to Elasticsearch via RulesClient.
4. We will have schemas of forms used on the Rule Creation/Editing pages.

It will be a successful refactoring if, given the refactored schemas:

• It is easy to add a new common or rule type-specific parameter.
• It is easy to add a new rule type.
• Adding them does not require changing a lot of files.
• Adding them is low risk.
• There are no duplicated schemas.
• There is no copy-paste of fields in schemas and functions.
• When passing a rule from one function to another, the rule is passed as a whole instead of passing its destructured parameters.

Details

Schemas we have

Backend level:

• Alerting Framework rule types and params:
  • security_solution/server/lib/detection_engine/schemas/rule_schemas.ts
  • security_solution/server/lib/detection_engine/schemas/rule_converters.ts

HTTP API level:

• common types (domain model):
  • security_solution/common/detection_engine/schemas/common/schemas.ts
  • security_solution/common/detection_engine/schemas/response/rules_schema.ts
  • security_solution/common/detection_engine/schemas/request/rule_schemas.ts
  • security_solution/common/detection_engine/schemas/request/add_prepackaged_rules_schema.ts
  • security_solution/common/detection_engine/schemas/request/import_rules_schema.ts
• request-response types:
  • security_solution/common/detection_engine/schemas/request/*
  • security_solution/common/detection_engine/schemas/response/*

Frontend level:

• common types (domain model):
  • security_solution/public/detections/containers/detection_engine/rules/types.ts
• request-response types:
  • security_solution/public/detections/containers/detection_engine/rules/types.ts
  • security_solution/public/detections/containers/detection_engine/rules/api.ts

Sub-tasks

• [Security Solution][Detections] Adds framework for replacing API schemas #82462
• [Security Solution][Alerts][Rules] Force optional fields to be accounted for in conversion to rule response schema #136910 (PR)
• Get rid of the dynamic common/detection_engine/schemas/response/rules_schema.ts (PR)
• Consolidate request/rule_schema* and response/rule_schema* files. Move most of request/rule_schemas.ts to common/rule_schemas.ts and better separate the "base" and "type specific" schema logic from create, update, response, etc. This way we can keep request and response schemas organized into their own sections, and also remove the dependency of response on request/rule_schemas.ts. (PR)
• [Security Solution] Get rid of the OrUndefined types #138614 (PR)
• Get rid of the frontend rule schema
• Get rid of the Cypress-specific rule interfaces like CustomRule (comment)
• [Security Solution] Rule filters: strict schema #147438
• [Security Solution] Rule filters: saved object migrations #147441
• TBD

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)