[Stack Monitoring] Support for integrations

[neptunian]

Summary

The Stack Monitoring-supported integration packages (elasticsearch, kibana, logstash) are currently in an experimental state. While Stack Monitoring added support to read from the datastreams created by these integrations in 8.1, there are remaining issues that prevents the packages from being an official source of metrics powering the SM UI.

This issue tracks the effort to bring these integration packages to a GA state, providing feature parity with the 8.x beats modules running in standalone mode in the Stack Monitoring UI.

Useful links:

• elastic-package repo
• package-storage repo
• package-registry repo
• dev documentation for working with packages

(1) Initial functionality

Utils

• Remote dev solution for elastic integrations integrations#3849
• [Stack Monitoring] Create pre-configured elastic-package environment #138052
• [Stack Monitoring] Install scripts for integration packages integrations#4061

Packages

• [Stack Monitoring] Align integration packages mappings with metricbeat mappings #135379
• [Stack Monitoring] update data streams dataset section integrations#3929
• [Stack Monitoring] Support for error documents integrations#4011
• [Stack Monitoring] Fix kibana package logo integrations#4297
• [Stack Monitoring] Align integration packages ingest pipelines with filebeat #137112
• [Stack Monitoring] Fix elasticsearch package logo #138865
• [Stack Monitoring] remove release field integrations#4107
• [Stack Monitoring] create enterprise search package #143263

Stack Monitoring

• [Stack monitoring] Add agent-compatible data stream patterns to data fetching patterns #104271
• [Stack Monitoring] update index patterns with filebeat-* to use integration logs-* #120416
• [Stack monitoring] Add data stream patterns to monitoring health endpoint #138864
• [Stack Monitoring] Update metrics-* index pattern #135382
• [Stack Monitoring] Agent fails to publish docs to metrics-elasticsearch.stack_monitoring.enrich index #140039
• [Stack Monitoring] Removing remaining usage of legacy getIndexPatterns() #123508
• [Stack Monitoring] Update elasticsearch.pending_tasks .monitoring mappings #141305
• [Stack Monitoring] Update metricbeat's elasticsearch mappings beats#33211
• [Stack Monitoring] Investigate reading apm-server metrics from agent indice #143265
• [Stack Monitoring] Support for apm-server/beats Agent subprocesses #144701

(2) Testing

• [Stack Monitoring] Add system tests integrations#4008
• [Stack Monitoring] Support automatic update of stack version dependencies integrations#4013
• https://github.com/elastic/observability-dev/issues/1660
• [Stack Monitoring] Validate integrations on standalone agent #137195

(3) Documentation

• [Stack Monitoring] Docs pages for agent-based monitoring observability-docs#2602
• [Stack Monitoring] user stories for migration to Agent #120414
• [kibana] Mention elasticsearch data dependency in readme integrations#3974

(4) Release

• [Stack Monitoring] Promote packages to production integrations#3917

(5) Clean up

• [Stack Monitoring] Telemetry support for metrics-* agent data #123869
• [Stack Monitoring] clean filebeat processors integrations#4187
• [Stack Monitoring] Collect metrics-* errors in health API #140358
• [Stack Monitoring] Update metricbeat's elasticsearch mappings beats#33211

Acceptance criteria:

• Stack Monitoring UI is enable-able by installing elastic-agent and the Elasticsearch/Kibana/Logstash packages
• Stack Monitoring UI and alerts behave similarly when reading data from metricbeat or from the packages
• Troubleshooting documentation exists to provide guidance in onprem/cloud issues
• A dashboard provides insights in usage

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[matschaffer]

@crespocarlos @klacabane I was just noticing we don't have integrations (I think) that would cover:

• beats
• apm server (integration server)
• enterprise search

Those 3 components can be activated in the Stack Monitoring UI, so I'm not sure we can recommend migrating to agent/package based until we have packages for those.

[crespocarlos]

Yeah. Fair point. I don't think we have either.

I'm wondering if we could make Kibana, Logstash and ES packages production ready and start with them - perhaps they can be tagged as beta for now. The SM UI will work with both methods anyways. We can't surely recommend phasing out metricbeat for SM without having all the packages, but users could start using the packages.

The remaining packages will gradually be released and, once we're done, we'll change the documentation to recommend the migration.

[matschaffer]

Per discussion yesterday, beats and apm-server might be covered by the agent package. In an agent-based setup we probably don't need a beats monitoring package.

So that'd just leave enterprise search on the list of things that Stack Monitoring supports today that we don't have observability packages for.

[miltonhultgren]

Does that also apply for APM Server running outside of Elastic Agent?

[matschaffer]

What I got from @klacabane is that we might be able to assume standalone apm server doesn't exist in the agent/package-based world.

Current stack-monitoring for standalone apm server can continue to be done using standalone metricbeat (or probably even internal collection for some time).

[klacabane]

I think this boils down to how we want the transition to agent to happen and if we want the SM experience to remain the same as a first step of the migration, or if we're fine moving into integration territory earlier and breach the parity milestone that this ticket aims at. For elasticsearch/kibana/logstash the answer is yes, and I would add enterprise search to that list. The reason is that we have clear plans for these with Platform Observability. For beats and apm server I'm not sure:

• Beats is an implementation detail in agent world and I don't think we want the agent abstraction to be leaky, mostly because it can confuse users and beats will be eventually deprecated so this is a good opportunity to already get rid of the concept. So here what we could do is to detect whether metrics are collected by agent and offer a redirection to the Agent integration (maybe a dashboard if we have a reference) under the stack monitoring Beats section
• in both standalone/fleet cases the Apm integration needs to be installed so any dashboards that it contains will be available. We can have a mechanism similar to the beats one and take the opportunity to jump over the SM parity step. Alternatively the parity step could probably be achieved by adding the apm indices in stack monitoring if the documents are similarly shaped, so that's low effort and I'm fine with that as well

[klacabane]

After looking at apm-server it looks like the metrics are not automatically collected by agent when running as a subprocess and needs a separate metricbeat to collect them. For the beats subprocesses, the metrics are collected at metrics-elastic_agent.(metricbeat|filebeat)-*. One thing I forgot is that beats are not only filebeat and metricbeat spawned by agent but could also be independant heartbeat, auditbeat.. that may not have a corresponding entry in the agent catalog so we should still be able to monitor these with agent.

So for beats, including apm-server, it looks like we need two things:

• read from metrics-elastic_agent.*beat-* to surface metricbeat and filebeat spawned by agent
• create a beats package that collects metrics of independant beats and apm-server

We could also investigate if agent would be able to collect apm-server metrics itself just like it does for the beats

---

Digging a bit further I see references of metrics-elastic_agent.apm_server-*, metrics-elastic_agent.heartbeat-* ... so everything should already be there and it would just be a matter of adding these indices to the right places in Stack Monitoring

[klacabane]

Closing as all tasks here are complete. Follow ups will be created directly on the board