Move connector types into own plugin and split ownership

[mikecote]

It feels like a good time to split the ownership of the existing connector types, so the alerting team doesn't sit in the middle of the case team's efforts to develop their suite of connector types. After @pmuellr wrote some great guidelines, it feels they are in good hands.

Things to consider:

• Action validators should be passed allow-list config utils [Alerting] action validators should be passed allow-list config utils #64659
• Move connector types to the new plugin
• Split test suites
• Split CODEOWNERS, so the alerting team isn't an owner of anything related to the case connector types
• Come up with a list of connector types that will transition to the case team
• Move issues from our roadmap related to those connector types and re-assign to the case team

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[mikecote]

We will need an owner on alerting side for this meta issue as it makes sense to start work on this soon and have this broken down. Maybe in a few weeks once we are caught up with our documentation.

[mikecote]

The same will probably apply to SDHs for the first point of contact.

[gmmorris]

Hey @mikecote and @cnasikas ,
Given our conversation from Monday, is there any reason we don't move the issues relating to the incident management connectors to be under the Cases team?

I'm thinking of these issues:

1. [Alerting] JIRA connector to append comments to existing tickets #77319
2. [Actions] Jira action fails when newline symbol is in summary #104932
3. [Alerting] Jira and ServiceNow connectors UI should display API connection errors #105264
4. [Actions] ServiceNow implement support for adding custom incident fields. #78237
5. [Connectors] Add ServiceNow connector for Event Manger #94376

I do not think this should be moved to Cases though, as I suspect work is needed by us here:

1. Ability to resolve ServiceNow, IBM Resilient and Jira incidents #83221

Any objections?

[cnasikas]

Hey @gmmorris! About the issues:

1. This is a feature not related to the Jira connector. The user asks if it is possible to update the incident in Jira when a second alert of the same rule fires. To my understanding, an alert needs to save some kind of state to provide that functionality, right?
2. This can be fixed on the connector's level. I moved it to Cases.
3. This can be fixed on the connector's level. I moved it to Cases.
4. This can be owned by Cases. It is a challenging request 😄
5. This can be owned by Cases. Again a challenging request 🙂

You are right about #83221

Thanks!

[gmmorris]

Ah, fair point about #77319!

We can keep that, thanks :D

[mikecote]

Regarding #83221, I'm not sure what platform enhancements would be needed to support such capabilities. It feels like the connector needs a linked list between "dedupKey" and external incident ID. Some R&D will find out the options but may be solvable without changing the platform.

The same mechanism could solve #77319.

[gmmorris]

I think @YulNaumenko looked into ServiceNow at the time and couldn't find a way to do this without some kind of state that carries across the reaction executions, but I'm not sure.

Definitely worth a detailed investigation... and I guess, ideally, we'd verify SN, IBM and Jira can't do it before we change the platform. Any thoughts @cnasikas ?

[pmuellr]

Trying to solve the following in alerting will be hard:

• [Alerting] JIRA connector to append comments to existing tickets [Alerting] JIRA connector to append comments to existing tickets #77319
• Ability to resolve ServiceNow, IBM Resilient and Jira incidents Ability to resolve ServiceNow, IBM Resilient and Jira incidents #83221

Hard because we need to invent a mechanism to capture and persist the output of a connector execution somehow with the rule. And then make it available to the user creating the rule to somehow reuse it. All based on a particular alert instance, because of course a rule could schedule actions for multiple instances. It's also possible that you could add two Jira connectors (or same connector twice) to the same action group / alert instance (not sure of the use case for this, but ... maybe?), so the persisted state is actually alert instance / instance of action, and we don't even have a concept of "instance of action" to differentiate two Jira actions used in the same action group).

OTOH, in theory this could be done in cases, where they could capture that status and persist it in the case. Hopefully we could reuse our existing connector code, perhaps with some small tweaks to return the appropriate data if we're not already, and assuming we can already reuse the connector executors like that.

This would leave us in a world where these connectors used outside of connectors would not be auto-resolvable, or be able to "post to the same issue" kind of thing. You'd need to use cases for that. That seems fine to me technically, not sure what the product concerns would be though.

[cnasikas]

I think @YulNaumenko looked into ServiceNow at the time and couldn't find a way to do this without some kind of state that carries across the reaction executions, but I'm not sure.

Definitely worth a detailed investigation... and I guess, ideally, we'd verify SN, IBM and Jira can't do it before we change the platform. Any thoughts @cnasikas ?

With our new ServiceNow application, we could update an incident in ServiceNow by using the correlation_id. When you push an incident, the application will check if there is an issue with the provided correlation_id. If there is, the incident will be updated. So if we put correlation_id: {{ruleID}} then every time an alert fires it will update the incident (assuming the ruleID is unique and the same on every alert execution). @gmmorris @mikecote @pmuellr Do you want me to support it in our new integration?

I do not think there is a way to do it for Jira & IBM resilient without platform changes.

OTOH, in theory this could be done in cases, where they could capture that status and persist it in the case. Hopefully we could reuse our existing connector code, perhaps with some small tweaks to return the appropriate data if we're not already, and assuming we can already reuse the connector executors like that.

I think this is possible.

[mikecote]

So if we put correlation_id: {{ruleID}} then every time an alert fires it will update the incident (assuming the ruleID is unique and the same on every alert execution).

I would recommend something like {rule.id}:{alert.id} (what we do for PagerDuty's dedupKey) as a default, so the incidents are unique per alert. We also make this configurable in the rule action params for PagerDuty, so if the user wishes to have a single incident, they can change it to {rule.id}.

[cnasikas]

Thanks, @mikecote! So if I have it as {rule.id}:{alert.id} it will create an incident every time. If I have it as {rule.id} it will update the incident, right? I can do that. Are you ok with having this feature?

[mikecote]

So if I have it as {rule.id}:{alert.id} it will create an incident every time. If I have it as {rule.id} it will update the incident, right? I can do that. Are you ok with having this feature?

Yup that makes sense to me, it's important to have default behaviour of incident per alert ({rule.id}:{alert.id}) until we bring in the concept of the incident for the entire rule ({rule.id}) (#68828). But giving the users the ability to change it to {rule.id} manually works in the interim 👍 You can see how we did it for PagerDuty's dedupKey, it would be the same concept.

[mikecote]

Blocked on #64659.

[ymao1]

This should probably be split up into several steps:

• Move server side code from actions/server/built_in_action_types into new plugin - [Response Ops] [Connectors] Move connectors to own plugin #139867
• Move client side code from triggers_actions_ui/public/built_in_action_types into new plugin - [Response Ops] [Connectors] Move connectors type UI components to stack_connectors plugin #140444
• Split functional test suites - [Connectors] Split functional tests by connector type #141956