Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[System][Process] Add dimensions metadata; remove duplicated fields #6407

Merged
merged 3 commits into from
Jun 1, 2023
Merged

[System][Process] Add dimensions metadata; remove duplicated fields #6407

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8ae309b
remove duplicated fields for the process data_stream; add dimensions
tetianakravchenko May 31, 2023
11ba4b5
fix pr link
tetianakravchenko May 31, 2023
e9fdfca
resolve conflicts
tetianakravchenko Jun 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/system/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.31.0"
changes:
- description: Add dimension metadata to the process data_stream
type: enhancement
link: https://github.com/elastic/integrations/pull/6407
- version: "1.30.0"
changes:
- description: Add dimension fields to metrics all data_streams, except core, network and process to support TSDS migration
Expand Down
77 changes: 6 additions & 71 deletions packages/system/data_stream/process/fields/agent.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,21 @@

Examples: AWS account id, Google Cloud ORG Id, or other unique identifier.'
example: 666777888999
dimension: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tetianakravchenko The agent file we are adding here, is this shared across all integrations that have adopted dimensions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you mean if the content of this file is the same for all tsdb data_streams? no, there are less fields defined for cloud providers - #5193 (comment)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. But the same file is copy / pasted (manually) for the non cloud ones.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at least for data_streams within the same package I believe it should be the same. From what I've noticed fields folder structure is different for different packages. In general there might be some differences, like for example - in diskio there are some specific to this data_stream fields - https://github.com/elastic/integrations/blob/main/packages/system/data_stream/diskio/fields/agent.yml#L195-L205

- name: availability_zone
level: extended
type: keyword
ignore_above: 1024
description: Availability zone in which this host is running.
example: us-east-1c
dimension: true
- name: instance.id
level: extended
type: keyword
ignore_above: 1024
description: Instance ID of the host machine.
example: i-1234567890abcdef0
dimension: true
- name: instance.name
level: extended
type: keyword
Expand All @@ -42,12 +45,14 @@
ignore_above: 1024
description: Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean.
example: aws
dimension: true
- name: region
level: extended
type: keyword
ignore_above: 1024
description: Region in which this host is running.
example: us-east-1
dimension: true
- name: project.id
type: keyword
description: Name of the project in Google Cloud.
Expand All @@ -67,6 +72,7 @@
type: keyword
ignore_above: 1024
description: Unique container id.
dimension: true
- name: image.name
level: extended
type: keyword
Expand All @@ -90,12 +96,6 @@
ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.'
type: group
fields:
- name: architecture
level: core
type: keyword
ignore_above: 1024
description: Operating system architecture.
example: x86_64
- name: domain
level: extended
type: keyword
Expand All @@ -105,13 +105,6 @@
For example, on Windows this could be the host''s Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host''s LDAP provider.'
example: CONTOSO
default_field: false
- name: hostname
level: core
type: keyword
ignore_above: 1024
description: 'Hostname of the host.

It normally contains what the `hostname` command returns on the host machine.'
- name: id
level: core
type: keyword
Expand All @@ -121,64 +114,6 @@
As hostname is not always unique, use values that are meaningful in your environment.

Example: The current usage of `beat.name`.'
- name: ip
level: core
type: ip
description: Host ip addresses.
- name: mac
level: core
type: keyword
ignore_above: 1024
description: Host mac addresses.
- name: name
level: core
type: keyword
ignore_above: 1024
description: 'Name of the host.

It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use.'
- name: os.family
level: extended
type: keyword
ignore_above: 1024
description: OS family (such as redhat, debian, freebsd, windows).
example: debian
- name: os.kernel
level: extended
type: keyword
ignore_above: 1024
description: Operating system kernel version as a raw string.
example: 4.4.0-112-generic
- name: os.name
level: extended
type: keyword
ignore_above: 1024
multi_fields:
- name: text
type: text
norms: false
default_field: false
description: Operating system name, without the version.
example: Mac OS X
- name: os.platform
level: extended
type: keyword
ignore_above: 1024
description: Operating system platform (such centos, ubuntu, windows).
example: darwin
- name: os.version
level: extended
type: keyword
ignore_above: 1024
description: Operating system version as a raw string.
example: 10.14.1
- name: type
level: core
type: keyword
ignore_above: 1024
description: 'Type of host.

For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment.'
- name: containerized
type: boolean
description: >
Expand Down
7 changes: 7 additions & 0 deletions packages/system/data_stream/process/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
name: process.pgid
- external: ecs
name: process.pid
dimension: true
- external: ecs
name: process.parent.pid
- external: ecs
Expand All @@ -24,6 +25,9 @@
name: host.mac
- external: ecs
name: host.name
dimension: true
- external: ecs
name: host.hostname
- external: ecs
name: host.os.family
- external: ecs
Expand All @@ -48,3 +52,6 @@
external: ecs
- name: service.type
external: ecs
- external: ecs
name: agent.id
dimension: true
3 changes: 2 additions & 1 deletion packages/system/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1801,6 +1801,7 @@ If running as less privileged user, it may not be able to read process data belo
| Field | Description | Type | Unit | Metric Type |
|---|---|---|---|---|
| @timestamp | Event timestamp. | date | | |
| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword | | |
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | |
| cloud.availability_zone | Availability zone in which this host is running. | keyword | | |
| cloud.image.id | Image ID for the cloud instance. | keyword | | |
Expand Down Expand Up @@ -1836,7 +1837,7 @@ If running as less privileged user, it may not be able to read process data belo
| host.os.full.text | Multi-field of `host.os.full`. | match_only_text | | |
| host.os.kernel | Operating system kernel version as a raw string. | keyword | | |
| host.os.name | Operating system name, without the version. | keyword | | |
| host.os.name.text | Multi-field of `host.os.name`. | text | | |
| host.os.name.text | Multi-field of `host.os.name`. | match_only_text | | |
| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | | |
| host.os.version | Operating system version as a raw string. | keyword | | |
| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | | |
Expand Down
2 changes: 1 addition & 1 deletion packages/system/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 1.0.0
name: system
title: System
version: 1.30.0
version: 1.31.0
license: basic
description: Collect system logs and metrics from your servers with Elastic Agent.
type: integration
Expand Down