[Security Solution] Allow users to edit required_fields field for custom rules

[approksiu]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168

Summary

We need to expose the required_fileds field in the rule edit page and allow editing it.
The required_fields is an optional field.

Background

We want to allow users to easily specify the prerequisites for their custom rules.

User story

• As a user, I want to be able to specify required fields for my custom rules.

Acceptance criteria

• Rule edit page shows the required_fields field in the Define Rule section.
• User can add required fields.
• On the rule edit page, there is a tooltip explaining what this field does, (and a link to documentation?).
• When the Elastic prebuilt rule is duplicated - the required_fields value should be kept.

Release progress

• Initial implementation is done. (PR)
• Feature is covered with automated tests.
• Feature is fully implemented and considered by the development team as ready to be released.
• Exploratory testing is done and the feature is approved by @jpdjere
• Acceptance testing is done and the feature is approved by @approksiu and @ARWNightingale.
• API docs PR is merged
• Serverless documentation is written by @joepeeples. The PR is merged (docs ticket).
• ESS documentation is written by @joepeeples. The PR is approved and ready to be merged (docs ticket).
• Feature is released in Serverless.

Planned release date in Serverless: May 20th, 2024.
Planned release date in ESS: v8.15.0.

[jpdjere]

@approksiu @ARWNightingale

Is it possible to use the fields drop-down component for adding required fields?

Yes, we have to think about the design here. The required_fields field looks something like this:

            "required_fields": [
              {
                "ecs": true,
                "name": "event.type",
                "type": "keyword"
              },
              {
                "ecs": true,
                "name": "host.os.type",
                "type": "keyword"
              },
              {
                "ecs": true,
                "name": "process.args",
                "type": "keyword"
              },
              {
                "ecs": true,
                "name": "process.command_line",
                "type": "wildcard"
              },
              {
                "ecs": true,
                "name": "process.name",
                "type": "keyword"
              },
              {
                "ecs": true,
                "name": "process.parent.executable",
                "type": "keyword"
              },
              {
                "ecs": true,
                "name": "process.pe.original_file_name",
                "type": "keyword"
              }
            ]

so basically a collection of objects with 3 properties:

• name: just a string, I think this should be a free form text field - allowing only one word. We cannot offer a dropdown here since we don't know all the possible field values.
• type: I think this can be a dropdown. These are all the current values for type that our security-rules offer right now:

          {
            "key": "keyword",
            "doc_count": 15694
          },
          {
            "key": "unknown",
            "doc_count": 958
          },
          {
            "key": "long",
            "doc_count": 380
          },
          {
            "key": "ip",
            "doc_count": 301
          },
          {
            "key": "wildcard",
            "doc_count": 275
          },
          {
            "key": "boolean",
            "doc_count": 166
          },
          {
            "key": "match_only_text",
            "doc_count": 24
          },
          {
            "key": "text",
            "doc_count": 6
          }
        ]

so it's basically a great majority of keyword but some other. We can retrieve the whole list from https://github.com/elastic/kibana/blob/main/packages/kbn-field-types/src/types.ts , but note that our types are split between ES types and Kibana types so we'll have to merge those.

• ecs: just a boolean, true vs false. So we can have a dropdown or a checkbox.

So the design could look something similar to Reference URL:

with a button to add more required fields, but instead of one long text field, each line has the three fields mentioned above.

[approksiu]

@jpdjere I was thinking more something similar to alert suppression setup interface.

[approksiu]

Check the suppression component. Question: what to do if the field is mapped with different types in different indexes? Does the suppression component "know" if field is ecs-compliant and what type does it have?

[jpdjere]

@approksiu @ARWNightingale
I checked how the Alert Suppression component works, and we can reuse that functionality.

Basically, we call the [useRuleIndexPattern](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/pages/form.tsx#L128) hook to retrieve the list of fields, and the list looks like this:

[
    {
        "name": "@timestamp",
        "searchable": true,
        "type": "date",
        "aggregatable": true,
        "esTypes": [
            "date"
        ]
    },
    {
        "name": "_id",
        "searchable": true,
        "type": "string",
        "aggregatable": false,
        "esTypes": [
            "_id"
        ]
    },
    // [... more fields]
    {
        "name": "agent",
        "searchable": true,
        "type": "conflict",
        "aggregatable": false,
        "esTypes": [
            "text",
            "object"
        ],
        "conflictDescriptions": {
            "text": [
                "logs-ti_1"
            ],
            "object": [
                ".ds-auditbeat-8.3.3-2024.01.15-000001",
                ".ds-packetbeat-8.3.3-2024.01.15-000001"
            ]
        }
    },
    {
        "name": "agent.build.original",
        "searchable": true,
        "type": "string",
        "aggregatable": true,
        "esTypes": [
            "keyword"
        ]
    },
     // [... thousands fields more]
]

So we can prepopulate a dropdown field showing the fields name, and when the user selects it, we can prepopulate another dropdown to its right that contains the possible types for the fields, which can get get from each field's esTypes array:

The only missing data here is whether the field is ECS compliant or not. I don't think we need to display this to the user, we can calculate it on our side and save it without the user knowing about it. To calculate it, we can use the computeIsEcsCompliant method.

[approksiu]

@jpdjere awesome! I like that we can define if the field is ecs-compliant and not have the user fill it in. I assume if one type option is available - it will be automatically selected for the user.

[jpdjere]

I assume if one type option is available - it will be automatically selected for the user.

Yes, once the user chooses a required field, let's preselect the first available option in the type dropdown by default (but the user can still change it if he wishes, and if there is more than 1 available type)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[nikitaindik]

Hey @approksiu and @ARWNightingale. I have updated the sandbox with latest Required Fields changes. Would you mind taking a look and providing feedback for it?

Sandbox: https://nikitaindik-pr-180682-editable-required-fields.kbndev.co/kbn
Credentials: https://p.elstc.co/paste/U6uTAkz3#WW6hltDBqi++3e3Kzpi5bGiEHhKtAfKEnkRFsK6juZL

[nikitaindik]

@approksiu @ARWNightingale Here's some additional info about the feature and testing it in the sandbox.

The basic flow goes like this: first you specify your index patterns (or a data view), then you can select required fields for these index patterns.

I have created two indices in the sandbox deployment: i1 and i2.

• Index i1 has fields: one with type text and three with type text.
• Index i2 has fields: one with type keyword and two with type integer.

Since for field one both text and keyword types are available, the dropdown will show both options.
For other fields, only one type is available, so only one option will be shown and the dropdown will be disabled.

Warnings

If a field that is not present in the index pattern is selected, you will see a warning message.
This can happen in the following cases:

• You have specified an index pattern, selected a required field from this index pattern, and then removed this index pattern.
• The index doesn't yet exist. For example, you have installed a prebuilt rule but the data for it hasn't been ingested yet.
• The index was removed.
• The mappings for the index were changed and the field is no longer present.

In any of these cases, you'll see a general warning message above the form section. And then also a more specific warning message next to the field that is causing the issue.

The warnings won't prevent the user to submit the form.

ESQL and ML rules

Here's how available dropdown options are determined for different rule types:

For all rule types except ESQL and ML, we take the index patterns specified by the user and fetch thier mappings (fields and their types). Then we use these fields and types to populate the dropdowns.

For ESQL rules we parse index patterns out of the query since there's no explicit index pattern form field. We then fetch the mappings for these index patterns and use them to populate the dropdowns.

For ML rules, we don't show "required fields" at all. ML rules are a special case.

1. The concept of "required fields" is sort of handled during creation of the ML job itself, where the user specifies the fields that are required for the job to run.
2. In the ML rule creation/editing interface, we don't display the index patterns a rule operates on. So, even if we allowed specifying required fields, the user would need to check the ML job details to see the index patterns the job uses.
3. The ML job dropdown includes both existing and not-yet-created jobs. We can't get index patterns for jobs that don't exist yet, so we can't fill the dropdowns with fields and types.

Sample queries

Here are a few sample queries that can be used to test the feature:
ESQL: from i* [metadata _id, _index, _version]
EQL: any where two == "*"
Lucene: one: "value1" AND two > 10 AND three: "pattern*"

[ARWNightingale]

@nikitaindik this looks good to me.

[approksiu]

Thanks @nikitaindik ! Looks good!

[banderror]

@vgomez-el Please start exploratory testing when you can

[nikitaindik]

Hey @joepeeples! I've created a ticket for documenting the UI and API changes: elastic/security-docs#5131

[vgomez-el]

@banderror Just to let you know, I started the exploratory testing at the end of the week, but after asking @nikitaindik about some doubts, he told me that there are some last minute changes he needs to work in and push into the branch. I will delay the feature exploratory testing until those changes are finished and pushed to the branch/environment to avoid duplicating testing efforts.

[nikitaindik]

Hey team! Here's what we have left to do for Required Fields:

• First I need to merge in the latest changes from the main branch, resolve any conflicts and make sure the tests pass.
• Then @xcrzx will need to code review the PR. I might need to make some updates based on Dmitrii's feedback.
• Once the PR gets the approval @vgomez-el will do exploratory testing.
• Then we'll need @approksiu to acceptance test it.
• Lastly, @joepeeples will need to prepare the docs.

Current Serverless release target is Monday, May 6th. But given the remaining work, we might need to aim for the following release on May 13th. This means we would need to complete the review, testing and the docs by the end of the next week.

Does moving the release to May 13th work for everyone? If it does, I'll update the dates in this issue and in the docs issue. Please reply below. Thanks!

[vgomez-el]

@nikitaindik Moving it to the next release works for me. Just let me know when the environment is ready and I will perform the exploratory testing over the feature

[nikitaindik]

Hey team! Since I'm still working on the PR comments, Required Fields won't make it into the 13th May Serverless release. Rescheduling it to May 20th. I have updated the date in this ticket and in the API docs ticket.

@xcrzx @approksiu @joepeeples @banderror

[nikitaindik]

Hey folks! I have fixed all the issues, answered all the comments and have reopened the PR for review. I would appreciate if you can start reviewing it asap, since we want to release it on the 20th. Thanks!

@xcrzx @maximpn

[jpdjere]

@nikitaindik

Finished exploratory testing, looking good! Great work 👍
Also, thanks for the super detailed PR description.

[approksiu]

@nikitaindik acceptance testing done. Looks good. Thank you!

[nikitaindik]

Hey folks! 🎉 The main Required Fields PR and the API docs PR, have both been merged. Woohoo!

This means the feature is going into the Monday, the 20th Serverless release! 🚀

High-fives to everyone who helped with the review and testing! 🙌💖

[banderror]

The feature went live in Serverless on Tue, May 21st 🎉
Thanks to everyone involved! I'm closing this ticket. We'll track elastic/security-docs#5131 separately.