[System][Process] Add dimensions metadata; remove duplicated fields (#…

…6407) * remove duplicated fields for the process data_stream; add dimensions Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * fix pr link Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> --------- Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
elastic · Jun 15, 2023 · d214340 · d214340
1 parent c9ede83
commit d214340
Show file tree

Hide file tree

Showing 5 changed files with 21 additions and 73 deletions.
diff --git a/packages/system/changelog.yml b/packages/system/changelog.yml
@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.31.0"
+  changes:
+    - description: Add dimension metadata to the process data_stream
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/6407
 - version: "1.30.0"
   changes:
     - description: Add dimension fields to metrics all data_streams, except core, network and process to support TSDS migration

diff --git a/packages/system/data_stream/process/fields/agent.yml b/packages/system/data_stream/process/fields/agent.yml
@@ -13,18 +13,21 @@
 
         Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.'
       example: 666777888999
+      dimension: true
     - name: availability_zone
       level: extended
       type: keyword
       ignore_above: 1024
       description: Availability zone in which this host is running.
       example: us-east-1c
+      dimension: true
     - name: instance.id
       level: extended
       type: keyword
       ignore_above: 1024
       description: Instance ID of the host machine.
       example: i-1234567890abcdef0
+      dimension: true
     - name: instance.name
       level: extended
       type: keyword
@@ -42,12 +45,14 @@
       ignore_above: 1024
       description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
       example: aws
+      dimension: true
     - name: region
       level: extended
       type: keyword
       ignore_above: 1024
       description: Region in which this host is running.
       example: us-east-1
+      dimension: true
     - name: project.id
       type: keyword
       description: Name of the project in Google Cloud.
@@ -67,6 +72,7 @@
       type: keyword
       ignore_above: 1024
       description: Unique container id.
+      dimension: true
     - name: image.name
       level: extended
       type: keyword
@@ -90,12 +96,6 @@
     ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.'
   type: group
   fields:
-    - name: architecture
-      level: core
-      type: keyword
-      ignore_above: 1024
-      description: Operating system architecture.
-      example: x86_64
     - name: domain
       level: extended
       type: keyword
@@ -105,13 +105,6 @@
         For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.'
       example: CONTOSO
       default_field: false
-    - name: hostname
-      level: core
-      type: keyword
-      ignore_above: 1024
-      description: 'Hostname of the host.
-
-        It normally contains what the `hostname` command returns on the host machine.'
     - name: id
       level: core
       type: keyword
@@ -121,64 +114,6 @@
         As hostname is not always unique, use values that are meaningful in your environment.
 
         Example: The current usage of `beat.name`.'
-    - name: ip
-      level: core
-      type: ip
-      description: Host ip addresses.
-    - name: mac
-      level: core
-      type: keyword
-      ignore_above: 1024
-      description: Host mac addresses.
-    - name: name
-      level: core
-      type: keyword
-      ignore_above: 1024
-      description: 'Name of the host.
-
-        It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.'
-    - name: os.family
-      level: extended
-      type: keyword
-      ignore_above: 1024
-      description: OS family (such as redhat, debian, freebsd, windows).
-      example: debian
-    - name: os.kernel
-      level: extended
-      type: keyword
-      ignore_above: 1024
-      description: Operating system kernel version as a raw string.
-      example: 4.4.0-112-generic
-    - name: os.name
-      level: extended
-      type: keyword
-      ignore_above: 1024
-      multi_fields:
-        - name: text
-          type: text
-          norms: false
-          default_field: false
-      description: Operating system name, without the version.
-      example: Mac OS X
-    - name: os.platform
-      level: extended
-      type: keyword
-      ignore_above: 1024
-      description: Operating system platform (such centos, ubuntu, windows).
-      example: darwin
-    - name: os.version
-      level: extended
-      type: keyword
-      ignore_above: 1024
-      description: Operating system version as a raw string.
-      example: 10.14.1
-    - name: type
-      level: core
-      type: keyword
-      ignore_above: 1024
-      description: 'Type of host.
-
-        For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.'
     - name: containerized
       type: boolean
       description: >

diff --git a/packages/system/data_stream/process/fields/ecs.yml b/packages/system/data_stream/process/fields/ecs.yml
@@ -6,6 +6,7 @@
   name: process.pgid
 - external: ecs
   name: process.pid
+  dimension: true
 - external: ecs
   name: process.parent.pid
 - external: ecs
@@ -24,6 +25,9 @@
   name: host.mac
 - external: ecs
   name: host.name
+  dimension: true
+- external: ecs
+  name: host.hostname
 - external: ecs
   name: host.os.family
 - external: ecs
@@ -48,3 +52,6 @@
   external: ecs
 - name: service.type
   external: ecs
+- external: ecs
+  name: agent.id
+  dimension: true
diff --git a/packages/system/docs/README.md b/packages/system/docs/README.md
@@ -1801,6 +1801,7 @@ If running as less privileged user, it may not be able to read process data belo
 | Field | Description | Type | Unit | Metric Type |
 |---|---|---|---|---|
 | @timestamp | Event timestamp. | date |  |  |
+| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword |  |  |
 | cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword |  |  |
 | cloud.availability_zone | Availability zone in which this host is running. | keyword |  |  |
 | cloud.image.id | Image ID for the cloud instance. | keyword |  |  |
@@ -1836,7 +1837,7 @@ If running as less privileged user, it may not be able to read process data belo
 | host.os.full.text | Multi-field of `host.os.full`. | match_only_text |  |  |
 | host.os.kernel | Operating system kernel version as a raw string. | keyword |  |  |
 | host.os.name | Operating system name, without the version. | keyword |  |  |
-| host.os.name.text | Multi-field of `host.os.name`. | text |  |  |
+| host.os.name.text | Multi-field of `host.os.name`. | match_only_text |  |  |
 | host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword |  |  |
 | host.os.version | Operating system version as a raw string. | keyword |  |  |
 | host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword |  |  |

diff --git a/packages/system/manifest.yml b/packages/system/manifest.yml
@@ -1,7 +1,7 @@
 format_version: 1.0.0
 name: system
 title: System
-version: 1.30.0
+version: 1.31.0
 license: basic
 description: Collect system logs and metrics from your servers with Elastic Agent.
 type: integration