[APM] remove processor.event field requirement for data streams

[axw]

Describe the feature:

I would like to remove processor.event as a requirement for logs, metrics, and errors; ideally also for transactions and spans, if possible.

Instead of requiring processor.event, the UI can use a boolean should filter with minimum_should_match to query for documents that either have processor.event (for old pre-8.0 documents/indices) or that have relevant data_stream.* fields.

Describe a specific use case for the feature:

1. Simplify the data model & mappings, removing historical cruft
2. Reduce unnecessary storage overhead
3. Enable application metrics/logs to be written to non-APM specific data streams, and still show up in APM UI

(3) essentially means querying metrics like system/container/process CPU usage either from metrics-* without a filter on processor.event: metric, or from apm-* with a filter.

More details:

In 6.x and 7.x, APM documents were written to indices whose names users could control. We set processor.event fields in the documents to distinguish the event type -- transaction, span, error, metric, and log.

Since 8.0, APM documents have been written to data streams: traces-apm*, logs-apm*, and metrics-apm*. Users cannot change the prefix, they can only change the namespace. Specifically, events are routed as follows:

• transactions and spans go into traces-apm-<namespace> or traces-apm.rum-<namespace>
• errors go into traces-apm.error-<namespace>
• metrics go into metrics-apm*-<namespace> (there are numerous metric data streams)
• (application) logs go into logs-apm.app.<service>-<namespace>

The processor.event fields is thus mostly redundant. It is possible to infer the event type from the data_stream.* fields:

• data_stream.type: traces implies it's either a transaction or a span
• data_stream.type: metrics implies it's a metric
• (data_stream.type: logs) and (data_stream.dataset: apm.error) implies it's an error
• for all other data_stream.type: logs it's a log

[elasticmachine]

Pinging @elastic/apm-ui (Team:APM)

[dgieselaar]

@axw I'm not sure how easy this all is with e.g. custom dashboards?

[axw]

@dgieselaar which part is hard? Could you do a filter on (processor.event: metric) or (data_stream.type: metrics) for example?

I suppose it would be a mildly breaking change though.

[dgieselaar]

@axw:

@dgieselaar which part is hard? Could you do a filter on (processor.event: metric) or (data_stream.type: metrics) for example?

Yes, but this happens in so many places that it's probably not worth it to spend the time making this change, or the complexity from maintaining backwards compatibility. But we do have a carrot that we can offer: we don't use processor.name anywhere in the UI.

[axw]

So will we never remove processor.event? Otherwise, what would it take?

One of the main motivations behind this is to enable sending system, runtime, and application metrics to metrics-generic-default. Not necessarily that exact data stream name, but something created by the generic metrics data stream index template built into Elasticsearch.

[dgieselaar]

@axw I don't think the answer is never, it's just a pretty huge effort on our side for something of which the value is unclear to us. Maybe a pitch is a better venue?

[axw]

Fair enough! I did hesitate, but didn't realise it would be such a big effort. I'll reopen as a pitch.

[axw]

(Not going to open as a pitch, there should be a new project process soon.)