Eliminate fingerprinting in Prometheus TSDB input for Elastic 8.13 and above

[StephanErb]

The current Prometheus integration relies on a fingerprint dimension:

    - name: labels_fingerprint
      type: keyword
      dimension: true
      description: Autogenerated ID representing the fingerprint of labels object and includes query name

Now that elastic/elasticsearch#93564 is resolved in 8.13, this is no longer needed. Instead, labels.* can be marked as dimensions directly. Due to the locality sensitive hashing used in elastic/elasticsearch#98023 we can expect that elastic/elasticsearch#99747 will deliver storage savings over the current approach.

cc @felixbarny @tetianakravchenko

[felixbarny]

I agree this is unblocked now und would lead to better performance and storage reduction. @lalit-satapathy do you think we can prioritize this?

[lalit-satapathy]

Hi @felixbarny,

Do we know if now objects can be marked dimension now ? Dont see them in the dimension doc.

[felixbarny]

We don't even need that because we can just create a dynamic template for prometheus.labels.* that defines labels as a keyword field type with time_series_dimension set to true.

Something like this:

{
  "mappings": {
    "dynamic_templates": [
      {
        "labels": {
          "path_match": "prometheus.labels.*",
          "mapping": {
            "type": "keyword",
            "time_series_dimension": true
          }
        }
      }
    ]
  }
}

[tetianakravchenko]

For the remote_write datastream: as I understand it will be still required to keep the fingerprint processor for the list of metric_names - https://github.com/elastic/integrations/blob/main/packages/prometheus/data_stream/remote_write/elasticsearch/ingest_pipeline/default.yml#L9-L24

but for collector - all ingest pipeline can be removed
and for query - fingerprint processor can be removed

[tetianakravchenko]

We don't even need that because we can just create a dynamic template for prometheus.labels.* that defines labels as a keyword field type with time_series_dimension set to true.

hey @felixbarny can you please explain why we don't need it?

this is how mapping looks like now:

for field definition:

    - name: labels.*
      type: object
      object_type: keyword
      description: |
        Prometheus metric labels

adding dimension: true to the definition above should produce the expected dynamic template, but it is not supported now. When using:

    - name: labels.*
      type: object
      object_type: keyword
      dimension: true
      description: |
        Prometheus metric labels

I am getting an error when building package:

elastic-package-0.98.2 build -v
...
   1. file ".../go/src/github.com/elastic/integrations/build/packages/prometheus-1.16.0.zip/data_stream/collector/fields/fields.yml" is invalid: field "prometheus.labels.*" of type object can't be a dimension, allowed types for dimensions: constant_keyword, keyword, long, integer, short, byte, double, float, half_float, scaled_float, unsigned_long, ip

[felixbarny]

For the remote_write datastream: as I understand it will be still required to keep the fingerprint processor for the list of metric_names

That's right. But we could remove the fingerprinting of labels and only do a fingerprint of the metric names.

I am getting an error when building package:

Seems like there's a gap in the support in elastic-package and Fleet to support the dimension parameter for dynamic templates. But Elasticsearch and TSDB does support that. Could you create an issue for this? It seems like it should be an easy fix.

[lalit-satapathy]

Seems like there's a gap in the support in elastic-package and Fleet to support the dimension parameter for dynamic templates.

Yes, we need to mostly would need to go through the fleet enhancements for this. We did exactly the same for dynamic templates for metric types here.

[tetianakravchenko]

fleet issue - elastic/kibana#179649

@felixbarny @lalit-satapathy can you please forward it to the correct team?

[jsoriano]

adding dimension: true to the definition above should produce the expected dynamic template, but it is not supported now. When using:

    - name: labels.*
      type: object
      object_type: keyword
      dimension: true
      description: |
        Prometheus metric labels

I am getting an error when building package:

elastic-package-0.98.2 build -v
...
   1. file ".../go/src/github.com/elastic/integrations/build/packages/prometheus-1.16.0.zip/data_stream/collect

@tetianakravchenko please try with the "compact" syntax:

    - name: labels.*
      type: keyword
      dimension: true
      description: |
        Prometheus metric labels

[felixbarny]

Looks like all prerequisites for this issue are in place. Can we move forward with this now?

[lalit-satapathy]

Yes, we can take this up as the fleet issue is resolved. CC: @gpop63

[lalit-satapathy]

Since both testing by @gpop63 here and @harnish-elastic here has confirmed that the new capability only works 8.14 onwards, we should enable the same in packages stack dependancy.

FYI: The relevant Fleet change has gone in 8.14.0.

[lalit-satapathy]

@agithomas will be creating a separate meta issue to track other packages need such migration, which can be taken in the future. Please link the issue here.

[agithomas]

@agithomas will be creating a separate meta issue to track other packages need such migration, which can be taken in the future. Please link the issue here.

@lalit-satapathy , please find the link to the meta issue

[gpop63]

@felixbarny we are fine with removing the labels_fingerprint fields from these data streams even if they are GA?

[felixbarny]

As this is just an internal field that doesn't really provide value by itself, other than aiding the enablement of TSDB, I think that's fine.