[Alerts as Data] - Finding a home

[yctercero]

Summary

The existing implementation of alerts as data includes the ruleRegistry, and the alertsClient. These currently both live within the ruleRegistry plugin. The ruleRegistry began as a very stateful plugin and was subsequently updated to resemble more of a stateless library that provides a toolkit for registering rules with the alerting framework. The alertsClient exposes CRUD methods for interacting with the alerts generated by the registered rules.

As it stands, which index an alert is written to during rule execution is determined by which solution the rule is registered with. So a rule whose rule type is registered with security solution will have its alerts written to .alerts-security.alerts. However, there's an exception! Users, using the advanced settings, can specify a separate index they want to use. Living in a stateless library, the alertsClient does not have the ability to dynamically query for which index to use.

Right now we are hardcoding the alert index names based on the information we get - ie: which solution we're dealing with. This is clearly not ideal as any new solutions would need to go in and update the code.

Long story short, there are a few questions we'd like to discuss and kick off here:

• There are circumstances in which our clients need state, where should this state live?
  • Should the ruleRegistry store state?
  • Should the alerting framework store this state?
• Is the ruleRegistry the right place for the alertsClient?

Discussions for reference

#100705 (comment)
#100705 (review)

[jasonrhodes]

Some context about the framework vs library situation: the main goal that I think we should focus on when trying to be "not a framework" is that we shouldn't block access to Kibana Alerting Framework APIs. We don't want to provide "registerType", we want rule type authors to use that API directly, with decoration tools provided by this plugin.

That said, this plugin should feel free to engage in creating its own APIs, storing its own state, etc. So in that sense, it doesn't need to purely be a stateless library package.

Example:

const { initializeRuleTypeDecorator } = require('../rule_registry');
const alerting = require('../alerting');

const { ruleTypeDecorator } = initializeRuleTypeDecorator({
  // ... index name, mappings, other config
});

const myRuleTypeConfig = ruleTypeDecorator({
  // ... my rule type config
});

alerting.registerType(myRuleTypeConfig);

In this pseudo-code-ish example, initializeRuleTypeDecorator could accept values like registration context, mappings, etc. and use those values to initialize the component templates, instantiate RBAC set up so that this feature is connected to its registered alerts as data indices, etc.

I'm not exactly clear on how/when code authors will want to gain access to an RBAC-scoped Alerts Client, so I don't know if that should be returned from initializeRuleTypeDecorator or something different, but this is the very vague shape of what I've been imagining.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[weltenwort]

The description above talks about some advanced setting that can be used to change the index. Which is that?

And on a related note, when do we plan to restore the functionality of the xpack.ruleRegistry.index setting, which the alerts data client doesn't respect due to the hard-coded index list?

[dhurley14]

@weltenwort I am forgetting what was the reasoning for the customizability of that setting again? Given this is a system index we probably should not allow users to change the name, especially given we are trying to move away from index-level privileges which, given a custom setting, would force us to resort to searching with current user and not the kibana system user as the kibana system user would not have access to that custom user-provided index. These are the changes we needed to push into ES to grant the kibana system user access to the alerts indices, for context.

[jasonrhodes]

@dhurley14 there are two "settings" that I know about, I'll try to provide context for both so that anyone following along can understand the difference:

1. Security solution advanced setting: this is something the security solution will manage on its own which allows the "namespace" value of the index to be configured to be the space ID automatically, to maintain backwards compatibility for users who currently segment alerts by space.

2. Legacy multi-tenancy: in kibana.yml, users can specify the settings for kibana.index and/or xpack.task_manager.index which store Kibana entities in segmented indices so that multiple Kibanas can interact with the same ES source data without conflicting with each other. Those options are currently deprecated in Kibana, but available through 7.x. Early in the RAC process, we outlined that it would be good to provide this setting for users already doing this kind of multi-tenant segregation because it would have already segmented their rules this way, so alerts should follow suit.

For this second one, we earlier settled on a separate kibana.yml flag, something along the lines of "alert.index" but do something like if "alert.index" is "banana", we store those alerts in .alerts_banana-{registrationContext}.{datasetSuffix}-{namespace}.

Since that point, I don't know if we've fully considered a) the ramifications of allowing this and b) user demand for it (although it will be a strange experience for rules to be managed this way but alerts to flow across "tenants"). @dhurley14 / @yctercero can you outline how much pain something like this might cause for RBAC?

Also, I think we talked about this setting being related to a setting where the customer can read alerts as the current user instead of the system user + RBAC filters, but it's been a long time since anyone has mentioned that too.

Lastly, if the rules are already segmented by tenant, would the alerts those rules create already be visibility-restricted without needing this separate flag? I think the answer is no but my mind is mush and I can't remember why.