[Actions] System actions

[cnasikas]

Summary

Actions can be executed by rules when alerts are triggered. A user, at the creation of a rule, can define which actions will be executed. To define actions, users need to create connectors. Connectors contain information needed by the actions to execute. For example, they can contain credentials, instance URLs, or configuration options. Usually, connectors are used to communicate with external services like PagerDuty or ServiceNow. Automated actions like opening a case or running an OsQuery do not need any configuration. This document explores ways of supporting these kinds of actions in the actions’ framework. From now on they will be referred to as system actions.

High-level solution

A system action is an action that triggers Kibana workflows. For example, creating a case, running an OsQuery, running an ML job, or logging. System actions will be defined by developers with the use of the existing actions’ framework. By using the existing actions framework, system actions will get all features supported by the framework like retries, using the task manager, conditional actions, etc.

Registration

Solutions should be able to register their system action the same way they register action types through the actions registry. The definition of the system action will be almost identical to the actions. The key differences are:

• isSystemAction must be set to true
• The validation of config and secrets should be empty
• requiredPrivileges should be defined if authentication is needed (See RBAC section)

SavedObjects

System actions will not be persisted as a saved object as they do not have any configuration or secrets. As a consequence, all features supported by saved objects will not be available to system actions. For example, exporting/importing a system action will not be supported. A system action will be represented as a JS object by the actions’ framework. The object will have the same attributes as the actions’ saved object. Attributes, like the config and secrets, that are not needed by the system actions will default to empty values. This will simplify the actions’ framework and reduce the development effort.

The JS object will be loaded in memory at Kibana startup and it will live during the lifecycle of Kibana. The ID of a system action will be generated by the framework automatically and deterministically based on the actionTypeId for all Kibana nodes. The ID will never change during the lifecycle of Kibana and it will be the same between Kibana restarts.

APIs

Users will not be able to create, update, delete, and execute a system action through the APIs. The system actions will not be returned from the Get all connectors API, and the Get connector API and the system connector types will not be returned from the List connector types API.

Preconfigured connectors

Users will not be able to define system actions as preconfigured connectors. The action’s framework will throw an error on such an attempt.

Spaces

As with preconfigured connectors, the system actions will be space-agnostic. There is no need to create one connector for each space.

RBAC

It is possible for system actions to use kibana features that need their own authorization. For example, to use the case action the user needs to have “All” access to Cases to be able to create a case automatically. The actions framework will perform all the needed authorization checks on behalf of the system actions. Creators of system action will have to define a list of the Kibana privileges required by the system action with the use of security.authz.actions.*.get(...). For example, for the case action security.auth.actions.cases.get(‘case_owner’, ‘case_operation’) and for access to a specific saved object security.auth.actions.savedObject.get(‘my_so_object’,’my_operation’).

Regarding the UI, a user that does not have access to the feature privilege should not be able to view and select the system action when creating a rule. For example, if a user does not have access to Cases, they should not be able to select the case system action to automatically create cases even if she has permission to create rules.

Alerts

System actions, like the case action, may need information about the alerts generated by the rule. The framework should provide the alert IDs and indices to the system action through the executor function.

Execution

To preserve the stability of the system, users should not be able to configure the system action execution configuration (action parameters) through the Rules APIs, nor should they be able to modify it through UI forms. The alerting framework should dictate how the system actions will be executed. A system action will not have a group, a frequency, or an alertFilter. The only available and required properties will be id and params.

Action parameters

Usually users, with mustache templating through rule forms, pass certain parameters to the actions. System actions require certain data like a summarization of alerts, information about the rule, etc. To avoid setting them behind the scenes in the UI when configuring a rule and being part of rules’ action parameters and to keep the actions framework agnostic of the implementation details of the alerting framework and vice-versa, we will introduce Connector Adapters. With connector adapters action parameters set in the rule and parameters required by the system action can be different. A connector adapter, specified for a specific connector type, can build extra parameters, required by the connector, based on the rule parameters and extra metadata passed to it like the alerts or the rule.

The alerting framework will expose a registry to allow connectors to register their execution configuration.

PoC: #159866

Schema of actions in the rule

In the rules client (domain layer), a new attribute will be introduced called systemActions, which will aid the methods of the client to handle system actions. The scheme of the system action will be:

export interface RuleSystemAction {
  uuid?: string;
  id: string;
  actionTypeId: string;
  params: RuleActionParams;
}

The systemActions attribute will not be persisted to ES. All methods of the rules’ client will merge the default actions with the system actions before saving the rule. Also, they will split up the actions after they get any rule SO.

All rules APIs will not accept the systemActions in their requests. The system will deduct if an action is a system action automatically using the actions’ client. As system actions are not allowed to have a group, the property will be optional (at the moment it is required) in all APIs. The APIs will convert the actions to the schema needed by the rules client. Finally, the APIs will merge the system actions with the default actions before sending the response back to the client.

UI

Connectors
System actions do not require a configuration. For that reason, users will not be able to create or test system connectors from the UI. Also, the system actions will not be shown in the connector’s list on the connector’s page.

Rules
Users will be able to select a system action when creating a rule. The system actions will be shown in the same list as the other action types, and the users will not be able to select a system action twice. As with actions, the system actions will have to provide a React component to allow users to provide parameters for the system action, usually with the use of a form.

Tasks

• Create TS types for System actions ([Actions] Introduce the isSystemAction property to actions #160753)
• Allow registering system actions using the actionTypeRegistry ([Actions] Set system actions on Kibana start #160983)
• Prevent creating/updating/deleting/getting system actions from the APIs ([Actions] Set system actions on Kibana start #160983, [Actions] Do not return system actions from GET APIs #161845)
• Prevent creating system actions from the kibana.yml ([Actions] System actions enhancements #161340)
• Implement Connector Adapters ([Actions] Connector Adapters MVP #166101)
• Support system actions in the Rules APIs:
  • Create ([Alerting] Enable creating system actions through the Create Rule API #167884)
  • Bulk edit ([Alerting] Enable updating system actions through the Bulk Edit Rule API #168226)
  • Bulk disable ([RAM] System action in bulk disable api #170229)
  • Bulk enable ([RAM] System action in bulk enable api #170476)
  • Bulk delete ([RAM] System action in bulk delete #170741)
  • Update ([RAM] System actions update rule api #171099)
  • Legacy APIs ([System Actions] Complete API audit and update APIs to comply with System Actions #172937)
• Support RBAC ([Actions] System actions authorization #161341)
• Adapt UI to handle system connectors

Feature branch: #166267
Related: #155644, #70303, https://github.com/elastic/security-team/issues/6475, https://github.com/elastic/security-team/issues/5849, https://github.com/elastic/security-team/issues/6200, #153837

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[pmuellr]

Any chance we can leverage the internal implementation of preconfiguredConnectors, so we can maintain the system actions in that collection? Otherwise, we'll have to maintain a parallel collection of system actions, and pass it around / check it all the places we use preconfigured connectors as well.

Are these space agnostic? Perhaps they wouldn't be visible in a space due to RBAC, but otherwise are available in every space?

I'm not sure "no config and no secrets" is a must here. I don't see much of a need for secrets, but config would be a nice way to parameterize multiple system actions with one connector type.

---

Users will be able to use the execute API.

users will not be able to create or test system connectors from the UI

These seem to contradict each other. If they can use the execute API, not clear why you can't test from the UI, which is all that does.

[cnasikas]

Any chance we can leverage the internal implementation of preconfiguredConnectors, so we can maintain the system actions in that collection? Otherwise, we'll have to maintain a parallel collection of system actions, and pass it around / check it all the places we use preconfigured connectors as well.

I am not very familiar with the framework, but I think it is possible. I like the idea. It will make the code easier. Any suggestion for the naming? I think we should rename preconfiguredConnectors as it will not only contain preconfigured connectors.

Are these space agnostic? Perhaps they wouldn't be visible in a space due to RBAC, but otherwise are available in every space?

Yes, they will be available in every space. How preconfigured connectors work? Are available in all spaces?

I'm not sure "no config and no secrets" is a must here. I don't see much of a need for secrets, but config would be a nice way to parameterize multiple system actions with one connector type.

Multiple system connectors of the same action type will not be allowed. There will always be one connector for each type. For example, only one case connector will be available. Users will not be able to create a connector from the UI or the HTTP API. I can see a scenario where this unique connector needs to be parameterized based on some configuration specific to the deployment (kibana.yml). As the system connectors will not be persisted as a saved object and we will construct them in memory we can extend them in the future if needed. Wdyt?

These seem to contradict each other. If they can use the execute API, not clear why you can't test from the UI, which is all that does.

Maybe we can let the user test the connector from the UI but as system connectors do not have any form (they cannot be created from the UI) we should only show the testing tab in the UI.

[tomsonpl]

Hey, great to see progress with system actions, I am very excited about this functionality ❤️

I have some questions, to make sure we could easily migrate current 'response actions'.

• you mentioned RBAC permissions while configuring the rule, what about the permissions/license checks on executing the rule?
  eg. user has all in place when configuring the rule, but then downgrades the license to basic - will system actions be triggered?

• is it already decided how the system actions will get triggered, and would that be open for any configuration?
  Asking because we might have some actions depending on alert data, eg. osquery {{ mustache }} parameters in query, or {{ pid }} value in endpoint actions. Then I believe it could be worthy to calculate (limit) number of action calls to the possible minimum (for performance reasons) instead of calling an action per alert. I think that the rule query would be specific enough not tot rigger redundant alerts but there if a strange case happens where we have 5 alerts with pid 500, we don't have to call it 5 times.

• would it be possible to call multiple system actions of one type? eg. 1 attach to cases, but 5 different osquery in one rule?

Thanks in advance for the answers. Trying to address some issues as soon as possible :)
I love the current progress and I am very excited about the way we're going forward ❤️

[cnasikas]

Thank you @tomsonpl for the feedback! I also am very excited about this. I hope it will meet your needs.

The system actions will be supported by the actions' framework meaning that all functionalities of the framework will be used by the system actions out of the box. Any diverge from the framework is mentioned in this issue.

you mentioned RBAC permissions while configuring the rule, what about the permissions/license checks on executing the rule?
eg. user has all in place when configuring the rule, but then downgrades the license to basic - will system actions be triggered?

The actions framework does the appropriate permissions/license checks for all actions/connectors before executing. The same will apply to the system actions.

is it already decided how the system actions will get triggered, and would that be open for any configuration?
Asking because we might have some actions depending on alert data, eg. osquery {{ mustache }} parameters in query, or {{ pid }} value in endpoint actions. Then I believe it could be worthy to calculate (limit) number of action calls to the possible minimum (for performance reasons) instead of calling an action per alert. I think that the rule query would be specific enough not tot rigger redundant alerts but there if a strange case happens where we have 5 alerts with pid 500, we don't have to call it 5 times.

System actions, using connector adapters (see Action parameters section), will be able to define how the actions will behave. For example, you can define if you want the system action to be triggered with a summarization of alerts found in the last query or if you want the action to be triggered for each alert found. The configuration will be hidden by the users and they will not be able to change it. Based on the alert data the system action can decide how to proceed. As I am not familiar with the internals of the OsQuery and Endpoint actions I would like to discuss it more offline to be sure that our implementation takes care of your needs. cc @patrykkopycinski

would it be possible to call multiple system actions of one type? eg. 1 attach to cases, but 5 different osquery in one rule?

Yes, the behavior will be the same for system actions as with the "normal" connectors.

[cnasikas]

We decided for the MVP that system actions will always run with a summarization of the alerts. This behavior cannot be configured by connector adapters. Also, connector adapters will only be available for system actions. I updated the issue accordingly.

[mdefazio]

A few questions from a UI standpoint:
Do we have a sense of priority when selecting these alongside connector actions? Security separates these into 'Response Actions'. (Feels a bit like a separate step rather than an 'or', but maybe some minor tweaks would solve that) Do we want to promote these more than connector actions? Do we see the system actions list growing rapidly, where perhaps a tab or toggle between connector actions and system actions might make sense? Or a single list with perhaps a 'system' badge on the appropriate ones? Curious to get more thoughts around this.

A system action is an action that triggers Kibana workflows. For example, creating a case, running an OsQuery, running an ML job, or logging.

I wasn't clear what 'logging' meant here?

If system actions are classified as those not needing configuration, what happens if I select Osquery or run ML job, and I don't have any setup? Do we need to account for jumping off into that flow? (Sounded like a no on this last question based on your previous comments)

Do we have a sense of what choices are required to run an ML job? Are there more options needed than just picking the job? Am I selecting anything from the alert context? (maybe that wouldn't make sense, but just curious how much we need to account for in the UI)

Here's a screen-smash dropping Osquery form into our action block (and removing the action frequency and connector option. Does this (roughly) align with our thinking on how these should be presented alongside the connector actions?

[Zacqary]

Do we want to promote these more than connector actions? Do we see the system actions list growing rapidly, where perhaps a tab or toggle between connector actions and system actions might make sense? Or a single list with perhaps a 'system' badge on the appropriate ones?

I'd prefer the second approach, and I'd even question whether we need the "System" badge. The user shouldn't need to care if there's a connector involved, they just want to see a list of icons with stuff they can do. Whether something is a "system action" or "connector action" is purely our concern as developers.

See the example of the Log connector, which the user should never need to care about.

[cnasikas]

Whether something is a "system action" or "connector action" is purely our concern as developers.

From the user perspective, the difference is that they cannot create, edit, or see the connector on the connectors page. If there is no separation at all (even with a badge) users may wonder why they cannot see the system connectors or create them and get confused.

[Zacqary]

@cnasikas We're preparing to merge the feature branch containing the system actions framework. Note that this will not ship any actual system actions, just the framework for them.

I understand the first planned system action is the Case Connector. Is this feature being tracked in #94498, or is there a new issue for implementation? I'd like to close this issue when the feature branch merges, but I want to make sure the next steps are being tracked.

[cnasikas]

Hey @Zacqary. The Case action is being tracked at #153837 and the PR for it is #168369. Feel free to close this issue when you merge the feature branch! Thank you so much for all the effort you put into this.

[tomsonpl]

This is exciting 🥇