Enable TSDB for System integration

[joshdover]

We need to enable TSDB indexing for the system integration's metric streams to reduce storage costs. This should be done in 3 broad phases:

• Update mappings to specify dimensions
• Update mappings to specify metric types
• Flip the toggle to enable TSDB indexing mode, update constraint.kibana.version to 8.8

You can find related issues and examples for other integrations in #5233

Current PRs:

• beta release PR - [System] Enable tsdb by default for beta testing for all but core, network and process data_streams #6427
• metric_type - add missing metadata - [System] Add missing metric_type metadata #6395
• dimensions network data_stream: [System] [Network] Add dimensions and clean up duplicated fields definition #6405
• dimensions for add data_sreams, except core, network, process - [System] Add dimensions to system package metrics data_streams only, except core data_streams #6118
• dimensions process data_stream: [System][Process] Add dimensions metadata; remove duplicated fields #6407
• core dimensions - [System][Core] Add dimensions for the core data_stream #6454
• enable tsdb on network - [System][Network] Enable TSDB for network data_stream #6469
• process metric_type [System][Process] Add metric_type metadata for the process.cgroup metrics #6493
• process add missing metric_types [system][process] Add metric_type metadata for object fields, set stack restriction to 8.9.0 version #7660
• enable TSDB on process - [system][process] Enable tsdb for process datastream #7672
• enable TSDB on core - [System][core] Enable tsdb #8637

Found issues:

• missing time_series_metric [Fleet] Mapping does not reflect the fields definition in package kibana#160403
• TSDB: numbers that are mapped as keyword are not validated correctly at routing time elasticsearch#96552

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[jlind23]

cc @andresrc for awareness / prio

[jlind23]

Assigning this to @lalit-satapathy per our email discussion.

[lalit-satapathy]

@tetianakravchenko will help do the TSDB enablement on system package.

Based on the analysis, the scope of the TSDB enablement on system package would be:

• dimensions present only in 3 out of 17 data_streams: memory, cpu and load. See any missing dimensions. Dimension need to be added to all remaining data_streams and testing.
  Prior work by @ruflin.
• metic_type present in diskio, core, memory, socket_summary, cpu, network,filesystem, load, fsstat, process_summary, uptime and process. We need to see if this work before is complete or anything pending here.
  Prior work.

As usual, we we separate the dimension work from new metric type work(if any) and track in separate PRs.

[tetianakravchenko]

Relevant for TSDB datastreams:

Datastream: core (metrics)

• [WIP] dimensions: will add it in separated PR, core.id should be used as an extra dimentions, when adding it, all documents are dropped - need investigate.

Datastream: cpu (metrics)

• dimensions: adjust according to the new list - [System] Add dimensions to system package metrics data_streams only, except core data_streams #6118

Datastream: diskio (metrics)

• dimensions: [System] Add dimensions to system package metrics data_streams only, except core data_streams #6118

Datastream: filesystem (metrics)

• dimensions: [System] Add dimensions to system package metrics data_streams only, except core data_streams #6118

Datastream: fsstat (metrics)

• dimensions: [System] Add dimensions to system package metrics data_streams only, except core data_streams #6118

Datastream: load (metrics)

• dimensions: adjust according to the new list - [System] Add dimensions to system package metrics data_streams only, except core data_streams #6118

Datastream: memory (metrics)

• dimensions: adjust according to the new list - [System] Add dimensions to system package metrics data_streams only, except core data_streams #6118

Datastream: network (metrics)

• dimensions: [System] Add dimensions to system package metrics data_streams only, except core data_streams #6118

Datastream: process (metrics)

• dimensions: [System] Add dimensions to system package metrics data_streams only, except core data_streams #6118

Datastream: process_summary (metrics)

• dimensions: [System] Add dimensions to system package metrics data_streams only, except core data_streams #6118

Datastream: socket_summary (metrics)

• dimensions: [System] Add dimensions to system package metrics data_streams only, except core data_streams #6118

Datastream: uptime (metrics)

• dimensions: [System] Add dimensions to system package metrics data_streams only, except core data_streams #6118

next datastream are skipped:

• application (input log: winlog)
• auth (input type: logfile)
• security (input type: winlog)
• syslog (input type: logfile)
• system (input type: winlog)

[lalit-satapathy]

Hi @joshdover, @jlind23,

We have the initial PR ready (with dimension work) as part of the TSDB enablement. Can someone help review the changes from the package owner perspective and also need help answering one query here. Thanks!

[tetianakravchenko]

Testing:

Before TSDB:

Disk used is missing for the time range < 1h

Disk used is missing for the time range < 1h

Swap usage - no value

Enable TSDB

Diskio:

- name: host
  type: group
  fields:
    - name: disk.read.bytes
      type: scaled_float
      unit: byte
      metric_type: gauge
      description: |
        The total number of bytes read successfully in a given period of time.
    - name: disk.write.bytes
      type: scaled_float
      unit: byte
      metric_type: gauge
      description: |
        The total number of bytes write successfully in a given period of time.

no time_series_metric for scaled_float in mapping:

mapping:

      "host": {
        "properties": {
          "architecture": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "containerized": {
            "type": "boolean"
          },
          "disk": {
            "properties": {
              "read": {
                "properties": {
                  "bytes": {
                    "type": "long"
                  }
                }
              },
              "write": {
                "properties": {
                  "bytes": {
                    "type": "long"
                  }
                }
              }
            }
          },

Created issue:

• missing time_series_metric [Fleet] Mapping does not reflect the fields definition in package kibana#160403
• enablement of tsdb on core data_stream is blocked by TSDB: numbers that are mapped as keyword are not validated correctly at routing time elasticsearch#96552
• [ ]

[lalit-satapathy]

@tetianakravchenko, Anything pending on this issue or it can be closed?

[tetianakravchenko]

@lalit-satapathy there is one more datastream to enable tsdb on - #8637. When merged this issue can be closed