Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update buckets type in metrics mappings to nested #97

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions docs/schema/observability/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,10 +105,10 @@ Simple Schema for Observability allows ingestion of both (OTEL/ECS) formats and

The Observability indices would follow the recommended for immutable data stream ingestion pattern using the [data_stream concepts](https://opensearch.org/docs/latest/opensearch/data-streams/)

Index pattern will follow the next naming template `sso_{type}`-`{dataset}`-`{namespace}`
Index pattern will follow the next naming template `ss4o_{type}`-`{dataset}`-`{namespace}`

**type**
- indicated the observability high level types "logs", "metrics", "traces" (prefixed by the `sso_` schema convention )
- indicated the observability high level types "logs", "metrics", "traces" (prefixed by the `ss4o_` schema convention )

**dataset**
- The field can contain anything that classify the source of the data - such as `nginx.access`
Expand All @@ -118,20 +118,20 @@ Index pattern will follow the next naming template `sso_{type}`-`{dataset}`-`{na

This strategy allows two degrees of naming freedom: dataset and namespace. For example a customer may want to route the nginx logs from two geographical areas into two different indices:

- `sso_logs-nginx-us`
- `sso_logs-nginx-eu`
- `ss4o_logs-nginx-us`
- `ss4o_logs-nginx-eu`

This type of distinction also allows for creation of crosscutting queries by setting the next index query pattern `sso_logs-nginx-*` or by using a geographic based crosscutting query `sso_logs-*-eu`.
This type of distinction also allows for creation of crosscutting queries by setting the next index query pattern `ss4o_logs-nginx-*` or by using a geographic based crosscutting query `ss4o_logs-*-eu`.

## Data index routing

The [ingestion component](https://github.com/opensearch-project/data-prepper) which is responsible for ingesting the Observability signals is responsible to route the data into the relevant indices.

The `sso_{type}-{dataset}-{namespace}` combination dictates the target index, `{type}` is prefixed with the `sso_` prefix into one of the supported type:
The `ss4o_{type}-{dataset}-{namespace}` combination dictates the target index, `{type}` is prefixed with the `ss4o_` prefix into one of the supported type:

- Traces - `sso_traces`
- Metrics - `sso_metrics`
- Logs - `sso_logs`
- Traces - `ss4o_traces`
- Metrics - `ss4o_metrics`
- Logs - `ss4o_logs`

For example if within the ingested log contains the following section:
```json5
Expand All @@ -146,7 +146,7 @@ For example if within the ingested log contains the following section:
}
}
```
This indicates that the target index for this observability signal should be `sso_traces`-`mysql`-`prod` index that follows uses the traces schema mapping.
This indicates that the target index for this observability signal should be `ss4o_traces`-`mysql`-`prod` index that follows uses the traces schema mapping.

## Observability Index templates

Expand All @@ -158,7 +158,7 @@ With the expectation of multiple Observability data providers and the need to co
## Observability Ingestion pipeline
The responsibility on an **Observability-ingestion-pipeline** is to create the actual `data_stream` in which it is expecting to ingest into.

This `data_stream` will use one of the Observability ready-made index templates (Metrics,Traces and Logs) and conform with the above naming pattern (`sso_{type}`-`{dataset}`-`{namespace}`)
This `data_stream` will use one of the Observability ready-made index templates (Metrics,Traces and Logs) and conform with the above naming pattern (`ss4o_{type}`-`{dataset}`-`{namespace}`)

**If the ingesting party has a need to update the template default index setting (shards, replicas ) it may do so before the actual creation of the data_stream.**

Expand Down
27 changes: 9 additions & 18 deletions schema/observability/metrics/metrics-1.0.0.mapping
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"index_patterns": [
"sso_metrics-*-*"
"ss4o_metrics-*-*"
],
"data_stream": {},
"template": {
Expand All @@ -27,14 +27,6 @@
"enabled": true
},
"dynamic_templates": [
{
"attributes_map": {
"mapping": {
"type": "keyword"
},
"path_match": "attributes.*"
}
},
{
"resources_map": {
"mapping": {
Expand Down Expand Up @@ -124,17 +116,14 @@
"observedTimestamp": {
"type": "date_nanos"
},
"value": {
"properties": {
"int": {
"type": "integer"
},
"double": {
"type": "double"
}
}
"value@int": {
"type": "integer"
},
"value@double": {
"type": "double"
},
"buckets": {
"type" : "nested",
"properties": {
"count": {
"type": "long"
Expand Down Expand Up @@ -176,6 +165,7 @@
"type": "long"
},
"positiveBuckets": {
"type" : "nested",
"properties": {
"count": {
"type": "long"
Expand All @@ -189,6 +179,7 @@
}
},
"negativeBuckets": {
"type" : "nested",
"properties": {
"count": {
"type": "long"
Expand Down
2 changes: 1 addition & 1 deletion schema/observability/traces/services-1.0.0.mapping
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"index_patterns": [
"sso_services-*-*"
"ss4o_services-*-*"
],
"template": {
"mappings": {
Expand Down
2 changes: 1 addition & 1 deletion schema/observability/traces/traces-1.0.0.mapping
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"index_patterns": [
"sso_traces-*-*"
"ss4o_traces-*-*"
],
"data_stream": {},
"template": {
Expand Down