diff --git a/rfcs/text/0039-tsdb-dimensions.md b/rfcs/text/0039-tsdb-dimensions.md new file mode 100644 index 0000000000..a600eb2c99 --- /dev/null +++ b/rfcs/text/0039-tsdb-dimensions.md @@ -0,0 +1,333 @@ +# 0039: TSDB Dimensions + + +- Stage: **0 (strawperson)** +- Date: **2023-04-11** + + + + + + + + + +## Fields + +This RFC proposes the annotating of certain ecs fields as `dimension`. This change is proposed to take the advantage of using `TSDB` offered by the elasticsearch without impacting the data injection. + +Annotating field as `dimension` is one of the important step in the process of TSDB adoption. Failing to annotate adequate number of fields as `dimension` when `TSDB` is enabled may lead to data loss. A large majority of fields that must be annotated as `dimension` fields are ecs fields. Presently, the Integration (Service Integration, Cloud Native, etc ) developers are expected to annotate ecs fields as `dimensions` in integration configuration. To avoid the duplicatation in configuration, minimize data loss probability, the RFC is proposed. `dimension` field takes two values - `true` and `false`. + + +Changes to :service mapping + +```yaml +--- +- name: service + title: Service + group: 2 + short: Fields describing the service for or from which the data was collected. + description: > + The service fields describe the service for or from which the data was collected. + + These fields help you find and correlate logs for a specific + service and version. + footnote: > + The service fields may be self-nested under service.origin.* and service.target.* + to describe origin or target services in the context of incoming or outgoing requests, + respectively. + However, the fieldsets service.origin.* and service.target.* must not be confused with + the root service fieldset that is used to describe the actual service under observation. + The fieldset service.origin.* may only be used in the context of incoming requests or + events to describe the originating service of the request. The fieldset service.target.* + may only be used in the context of outgoing requests or events to describe the target + service of the request. + reusable: + top_level: true + expected: + - at: service + as: origin + beta: Reusing the `service` fields in this location is currently considered beta. + short_override: Describes the origin service in case of an incoming request or event. + - at: service + as: target + beta: Reusing the `service` fields in this location is currently considered beta. + short_override: Describes the target service in case of an outgoing request or event. + type: group + fields: + + - name: address + level: extended + type: keyword + dimension: true + short: Address of this service. + description: > + Address where data about this service was collected from. + + This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). + example: 172.26.0.2:5432 + +``` +Changes to host mapping + +```yaml +--- +- name: host + title: Host + group: 2 + short: Fields describing the relevant computing instance. + description: > + A host is defined as a general computing instance. + + ECS host.* fields should be populated with details about the host on which + the event happened, or from which the measurement was taken. + Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes. + type: group + fields: + - name: name + level: core + type: keyword + dimension: true + short: Name of the host. + description: > + Name of the host. + It can contain what hostname returns on Unix systems, the fully + qualified domain name (FQDN), or a name specified by the user. + The recommended value is the lowercase FQDN of the host. + +``` +Changes to agent mapping + +```yaml +--- +- name: agent + title: Agent + group: 2 + short: Fields about the monitoring agent. + description: > + The agent fields contain the data about the software entity, if any, that collects, detects, or observes events on a host, or takes measurements on a host. + + Examples include Beats. Agents may also run on observers. ECS agent.* fields shall be populated with details of the agent running on the host or observer where the event happened or the measurement was taken. + footnote: > + Examples: In the case of Beats for logs, the agent.name is filebeat. For APM, it is the + agent running in the app/service. The agent information does not change if + data is sent through queuing systems like Kafka, Redis, or processing systems + such as Logstash or APM Server. + type: group + fields: + + - name: id + level: core + type: keyword + dimension: true + short: Unique identifier of this agent. + description: > + Unique identifier of this agent (if one exists). + + Example: For Beats this would be beat.id. + example: 8a4f500d +``` + +Changes to cloud mapping + +```yaml +--- +- name: cloud + title: Cloud + group: 2 + short: Fields about the cloud resource. + description: > + Fields related to the cloud or infrastructure the events + are coming from. + footnote: > + Examples: If Metricbeat is running on an EC2 host and fetches data from its + host, the cloud info contains the data about this machine. If Metricbeat + runs on a remote machine outside the cloud and fetches data from a service + running in the cloud, the field contains cloud data from the machine the + service is running on. + The cloud fields may be self-nested under cloud.origin.* and cloud.target.* + to describe origin or target service's cloud information in the context of + incoming or outgoing requests, respectively. However, the fieldsets + cloud.origin.* and cloud.target.* must not be confused with the root cloud + fieldset that is used to describe the cloud context of the actual service + under observation. The fieldset cloud.origin.* may only be used in the + context of incoming requests or events to provide the originating service's + cloud information. The fieldset cloud.target.* may only be used in the + context of outgoing requests or events to describe the target service's + cloud information. + reusable: + top_level: true + expected: + - at: cloud + as: origin + beta: Reusing the `cloud` fields in this location is currently considered beta. + short_override: Provides the cloud information of the origin entity in case of an incoming request or event. + - at: cloud + as: target + beta: Reusing the `cloud` fields in this location is currently considered beta. + short_override: Provides the cloud information of the target entity in case of an outgoing request or event. + type: group + fields: + - name: project.id + level: extended + type: keyword + dimension: true + example: my-project + short: The cloud project id. + description: > + The cloud project identifier. + Examples: Google Cloud Project id, Azure Project id. + + + - name: instance.id + level: extended + type: keyword + dimension: true + example: i-1234567890abcdef0 + description: > + Instance ID of the host machine. + + - name: provider + level: extended + example: aws + type: keyword + dimension: true + short: Name of the cloud provider. + description: > + Name of the cloud provider. Example values are aws, azure, gcp, or + digitalocean. +``` + +Changes to container mapping + +```yaml +--- +- name: container + title: Container + group: 2 + short: Fields describing the container that generated this event. + description: > + Container fields are used for meta information about the specific container + that is the source of information. + These fields help correlate data based containers from any runtime. + type: group + fields: + - name: id + level: core + type: keyword + dimension: true + description: > + Unique container id. +``` + + + + +## Usage + + + +Integration package development is the key beneficiary of this change. The fields of the document that are received from an integration receives a field mapping. If and when TSDB benefits are to be utilised, along with the field mapping with a metric type, at least one of the fields must receive `dimension: true` annotation. + +Example of field mapping in integrations with the field enabled as a dimension field. +```yaml +--- +- name: wait_class + type: keyword + dimension: true + description: Every wait event belongs to a class of wait event. + +``` +## Source data + +The source of this data comes from monitoring a host like a Linux machine, laptop or a k8s node. The can come delivered through different shippers like Elastic Agent system metrics inputs, apm agents, prometheus node exporter and other host metric collectors. + + + + + + +## Scope of impact + + + +## Concerns + +No concerns are known as of now. Presence of the `dimension:true` does not impact functionality. Elastic Stack version 8.7 is essential for this. + + + + + + +## People + +The following are the people that consulted on the contents of this RFC. + +* @agithomas | author +* @ruflin | subject matter expert +* @lalit-satapathy | reviewer +* @martijnvg | reviewer + + + +## References + +* [TSDB Design Document](https://github.com/elastic/elasticsearch-adrs/blob/master/analytics/tsdb/tsdb-design.md) +* [Oracle Package Pull Request for TSDB Migraiton](https://github.com/elastic/integrations/pull/4966) + + + +### RFC Pull Requests + + + +* Stage 0: https://github.com/elastic/ecs/pull/2172 + +