What activity from alerts and actions should be tracked in the event log?

[mikecote]

Let's use this issue to come up with data that should be captured into the event log that users would care about. This will give us a better idea how to design the UIs displaying such information and handle relationships if any.

There is basic work already underway to support the alert details page here: #55636. Though, understanding the bigger picture of what information we want to gather and how to display it will help us structure the the event log data #55640.

Example:

• Should the user be able to see their alert actions that failed to execute?
• Should the user be able to see how many attempts an alert action took before executing successfully?
• Should the user be able to see when the alert executor failed to run?

It's a fine line between gathering the information for the user versus gathering information for debugging. If it's mostly for debugging, we can use the server logs.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[peterschretlen]

There was an internal discussion around diagnostic APIs recently, and this came up:

"Actions" Alerting statistics breakdown by type: Email/Pagerduty/Slack/Webhook success & error counts at the connector level, if possible per error type (i.e. webhook error response codes and counts).

I feel like we are probably pretty close to being able to track this already. The only additional ask here was that we expose it as metrics/diagnostics through an API. A basic aggregation API might be a useful addition to the event log API.

cc @kostasb

[kostasb]

On top of logging these errors in server logs, an aggregation API for this activity will help the user compare outputs from points in time to quickly identify failing actions. As noted above, breakdown by error type by connector would be really useful for this.

[pmuellr]

Great examples, thx all!

We also have a need to do searches based on "visible alerts/actions" where visible means "RBAC indicates you can read this". Which complicates things. The event log only knows about saved object ids, not RBAC, so you need to do an alert/action query first to get application alert/action ids that are "visible", which would end up being a filter in the event log query.

But ideally, at an API level, it would be nice to allow some arbitrary DSL queries (or perhaps es sql would be good enough and easier) where we apply some "RBAC filter" internally, which would allow a pretty open-ended programatic API over the event log, across "visible" alerts/action/future-event-log-objects.

[ymao1]

Closing as we'd prefer to receive requirements rather than speculating what might be needed.