[ResponseOps] Visualize alerting metrics in Stack Monitoring

[chrisronline]

Relates to #122367

This is the final PR for the initial deliverable for the o11y of alerting initiative.

It involves using the indexed data from the previous work to show meaningful visualizations to the user within the Stack Monitoring UI. For now, we are adhering to existing design mechanisms within Stack Monitoring and simply adding to existing pages.

The metrics being added are:

• Rule Failures Rate
• Rule Executions Rate
• Action Failures Rate
• Action Executions Rate
• Rule Overdue Count
• Average Rule Overdue Delay
• Worst Rule Overdue Delay
• Action Overdue Count
• Average Action Overdue Delay
• Worst Action Overdue Delay

Testing

These metrics are only available through Metricbeat-based collection so testers will need to standup a local Metricbeat instance, configured to talk to their locally running Kibana.

I don't know about Metricbeat nightly binaries or how that works, so I just build from source - instructions on doing this are available in their docs.

Once you are able to build locally, you'll want to ensure you have the proper modules enabled:

CBR-MBP:metricbeat chris$ ./metricbeat modules list
Enabled:
elasticsearch-xpack
kibana-xpack

with configuration that points to your local environment:

- module: kibana
  xpack.enabled: true
  period: 10s
  hosts: ["https://localhost:5601"]
  username: "elastic"
  password: "changeme"
  ssl.verification_mode: none

(Note: You need to run the Elasticsearch module as well as the Stack Monitoring UI only works for Kibana data if there is also Elasticsearch data)

Start Metricbeat after this and visit the Stack Monitoring UI to verify data is properly flowing.

Screenshots

cc katefarrar (in the future)

[neptunian]

@elastic/response-ops Tests were unskipped and updated and now passing. Your approval is needed to merge.

[neptunian]

@elasticmachine merge upstream

[gmmorris]

@neptunian @XavierM looking at the original PR description, I get the feeling something is out of date.
Am I right in understanding that the metrics exposed by this PR in SM are:

• count of rule executions over time
• count of rule failures over time

Rather than the metrics displayed in the screenshot above?
If so, can we update the PR description to match what is actually being released? 🙏

Thank again, I know this has been a bit of a mess due to the reshuffle of this work across the teams.

[brianseeders]

Buildkite test this

[neptunian]

Buildkite test this

[neptunian]

@gmmorris The screenshots are correct. I've updated the PR description to list the metrics being added.

[neptunian]

Buildkite test this

[neptunian]

Buildkite test this

[neptunian]

Buildkite test this

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 2c6b946
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
monitoring	476.5KB	478.9KB	+2.4KB

History

• 💔 Build #50240 failed 7ca8600
• 💔 Build #50215 failed 92c7fa4
• 💚 Build #50175 succeeded fd0512e
• 💔 Build #50139 failed b1fc464
• 💚 Build #49175 succeeded 214c425

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @chrisronline