x-pack/plugin/otel: introduce x-pack-otel plugin #111091

gregkalapos · 2024-07-19T10:42:22Z

Solves #104731

This PR adds mappings for OpenTelemetry data via a new plugin otel-data

The PR is mainly based on:

the existing apm-data plugin from x-pack/plugin/apm: introduce x-pack-apm plugin #97546
the Add OpenTelemetry mappings #104455 draft PR by @felixbarny - most of the mappings are from there

What happens in the PR:

Created YamlTemplateRegistry as an abstract base class that loads mappings from YAML. The code of this class comes from APMIndexTemplateRegistry which now derives from YamlTemplateRegistry. The shared code is now moved to core.
Added otel-data plugin with OtelIndexTemplateRegistry which also derives from YamlTemplateRegistry.
Moved the mappings into YAML files from Add OpenTelemetry mappings #104455.

TODOs:

Discuss if we want to share some mappings between APM and OTel (candidates: all traces@ files from Add OpenTelemetry mappings #104455), those are currently not yet in the PR -> ~~Discussed: not at this point~~ Update: we did that - now some some basic things are shared across otel and apm.
Discuss data stream lifecycle for OTel mappings (right now, nothing is set in the PR) -> Discussed: Since these mappings are shared across multiple OTel use-cases (e.g. system metrics and APM, for now we don't set a default ILM/DSL, since it's hard to come up with a default. Using what APM does today may not be good for integrations - so for now no DSL is set in these templates).
Adding more tests to otel-data

x-pack/plugin/otel-data/src/main/resources/component-templates/ecs-tsdb@mappings.yaml

x-pack/plugin/otel-data/src/main/resources/component-templates/traces-otel@mappings.yaml

x-pack/plugin/otel-data/src/main/resources/component-templates/otel@mappings.yaml

x-pack/plugin/otel-data/src/main/java/org/elasticsearch/xpack/oteldata/OTelPlugin.java

x-pack/plugin/otel-data/src/main/resources/component-templates/metrics-otel@mappings.yaml

x-pack/plugin/otel-data/src/main/resources/component-templates/otel@mappings.yaml

strawgate · 2024-07-25T14:02:26Z

x-pack/plugin/otel-data/src/main/resources/index-templates/logs-otel@template.yaml

+  description: default logs template installed by x-pack
+  managed: true
+composed_of:
+  - logs@mappings


are we sure we want to include the component templates from the non-otel templates?

This would allow us to share very basic mappings and not duplicate those.

In this specific one, the only thing that we pull in is to set data_stream.type to "value": "logs" - the remaining things from logs@mappings are already covered in otel@mappings. I'd say setting data_stream.type to "value": "logs" is something which makes sense to share across log related templates.

I don't have much history on how often these mappings from core are changed - or if there would be any risk here, but I think this would be a good way to share these very basic mappings and also use it for OTel.

We discussed that potentially we could also share a few things for traces between otel and apm-data - that'd be also done via a mapping in core. But that's outside this PR.

we also are explicitly setting data_stream.type in the logs-otel@mappings though? and in this template?

That's an outcome from this discussion: #111091 (comment).

x-pack/plugin/otel-data/src/main/resources/component-templates/metrics-otel@mappings.yaml

x-pack/plugin/otel-data/src/main/resources/component-templates/traces-otel@mappings.yaml

x-pack/plugin/otel-data/src/main/resources/component-templates/logs-otel@mappings.yaml

x-pack/plugin/otel-data/src/main/resources/component-templates/otel@mappings.yaml

x-pack/plugin/otel-data/src/main/resources/component-templates/ecs-tsdb@mappings.yaml

.github/CODEOWNERS

x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_metic_tests.yml

felixbarny · 2024-07-30T13:42:31Z

@elasticmachine update branch

elasticmachine · 2024-07-30T13:42:34Z

expected head sha didn’t match current head ref.

x-pack/plugin/otel-data/src/main/resources/component-templates/otel@mappings.yaml

…YAML files

…/ecs-tsdb@mappings.yaml Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com>

gregkalapos · 2024-08-12T20:01:28Z

why are these being managed in yaml instead of matching the existing precedent of having them in json? I find yaml much more error-prone for unintentional changes, since it is heavily indentation reliant.

@dakrone that's my doing. We have historically had far more issues with business logic (wrong mappings, dynamic templates, etc.) than with syntax (indentation, "Norway problem", etc.). So we're optimising for readability and the ability to comment. Comprehensive (YAML) REST tests will cover basic syntax issues.

Yeah, so I took yaml, because apm-data already used yaml and we also discussed with @felixbarny and we said we'd move the json from #104455 to yaml. I'm personally fine with both, but my 2 cents is that apm and otel should use the same, because it'll be maintained by the same team. So with that I'd have a slight preference to move on with yaml for consistency.

There are already tests here that load these mappings and checks for them, so I'm less worried about accidental wrong indentation.

axw

LGTM! Thanks @gregkalapos 🙏 Just one minor comment.

@gregkalapos I think you raise a good question in https://github.com/elastic/elasticsearch/pull/111091/files/a6baf4644924ea4929ae2ab58ebac0b78f936e62#r1714175422 - it's also unclear to me if a user should be able to override the all_strings_to_keywords dynamic template. I can't think of any reasons.

If it shouldn't be possible to override, then I think it would be a good candidate for going into the index template. Same goes for top-level fields, but we can iterate on that in a followup.

x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs.tests.yml

@Custom

Add test to verify that @Custom template can add dynamic templates with a higher precedence

Remove summary_gauge and summary_counter since they are covered by summary_metrics

… registry

axw · 2024-08-14T02:27:34Z

@gregkalapos I see you removed summary_gauge & summary_counter, but summary_metrics doesn't have time_series_metric. Should we add that? I think with "counter"?

felixbarny · 2024-08-14T07:58:18Z

@gregkalapos I see you removed summary_gauge & summary_counter, but summary_metrics doesn't have time_series_metric. Should we add that? I think with "counter"?

I'm not sure if it makes much of a difference. I suppose it's relevant for downsampling. I'm not sure if counter or gauge is most appropriate here. The reason we removed summary_gauge & summary_counter is that there's no temporality for summary metrics. It's also not always monotonically increasing as it depends on the start and end timestamp. So you can effectively have either temporality without really having an indication of what the temporality is. At least that's my understanding of it @lahsivjar, please correct me if I'm wrong.

For now, I'm in inclined to just leave out time_series_metric and then potentially add one in a follow-up.

x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs.tests.yml

axw · 2024-08-14T09:17:17Z

I'm not sure if it makes much of a difference. I suppose it's relevant for downsampling. I'm not sure if counter or gauge is most appropriate here. The reason we removed summary_gauge & summary_counter is that there's no temporality for summary metrics. It's also not always monotonically increasing as it depends on the start and end timestamp. So you can effectively have either temporality without really having an indication of what the temporality is. At least that's my understanding of it @lahsivjar, please correct me if I'm wrong.

For now, I'm in inclined to just leave out time_series_metric and then potentially add one in a follow-up.

Sounds very reasonable to me.

…ec/test/20_logs.tests.yml Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com>

gregkalapos · 2024-08-14T11:32:35Z

I opened #111877 to collect follow up issues. Feel free to add to it if I missed something.

The new otel-data plugin is disabled by default. Since there are multiple things going in this PR - as discussed in other meetings - we should cut here and merge if this is ok to go. With that, other teams can test this more easily. #111877 will address remaining non-blockers.

x-pack/plugin/core/template-resources/src/main/resources/traces@settings.json

No lifecycle needed for OTel at this point

dakrone

LGTM for the Java parts, I can't speak to the content of the mappings (it looks like other Obs folks have already reviewed those parts)

gregkalapos · 2024-08-19T06:23:30Z

@elasticmachine update branch

* upstream/main: Fail `indexDocs()` on rejection (elastic#111962) Move repo analyzer to its own package (elastic#111963) Add generated evaluators for DateNanos conversion functions (elastic#111961) Clean the last traces from global retention in templates (elastic#111669) Fix known issue docs for elastic#111866 (elastic#111956) x-pack/plugin/otel: introduce x-pack-otel plugin (elastic#111091) Improve reaction to blob store corruptions (elastic#111954) Introduce `StreamingXContentResponse` (elastic#111933) Revert "Add 8.15.0 known issue for memory locking in Windows (elastic#111949)" Test get-snapshots API with missing details (elastic#111903) Add 8.15.0 known issue for memory locking in Windows (elastic#111949) # Conflicts: # server/src/main/java/org/elasticsearch/TransportVersions.java

@Custom

* Add YamlTemplateRegistry and OtelIndexTemplateRegistry with resource YAML files * Fix traces-otel template * Adding first yml tests * Base APMIndexTemplateRegistry on YamlTemplateRegistry * Update OTelPlugin.java * Update APMIndexTemplateRegistry.java * Update YamlIngestPipelineConfig.java * Adding traces tests * Update x-pack/plugin/otel-data/src/main/resources/component-templates/ecs-tsdb@mappings.yaml Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com> * Add mapper-version * Fix code-style * Rename `status.status_code` to `status.code` * Update otel@mappings.yaml Revert back to date due to missing support in ES|QL for date_nanos * Move dynamic_templates to metrics@mappings in core * Run gradlew :x-pack:plugin:core:spotlessApply * Update x-pack/plugin/otel-data/src/main/resources/component-templates/metrics-otel@mappings.yaml Co-authored-by: Carson Ip <carsonip@users.noreply.github.com> * Update 20_metic_tests.yml Workaround for TSDB timestamp issue: we push a custom template with higher priority and set time_series.start_time. * Update CODEOWNERS Adding obs-ds-intake-services as owner of the new otel-data plugin. Since we had some changes, also updating the owner of apm-data to the same team. * Change dynamic: strict to false * Skip "Reject invalid top level field" test * Update 20_metic_tests.yml * Add boolean as dimension test (skipping it for now) * Add booleans_to_keywords and enable corresponding test * Remove processor.event top level mapping Reason: for metrics and logs we can rely on the name of the datastream. For spans vs. transactions there are other fields we can use. * Remove booleans_to_keywords Because booleans are supported now as dimension on TSDB * Add alias service.language.name -> telemetry.sdk.language * cleanup * Update README.md * Update README.md * Update docs/changelog/111091.yaml * Move traces@settings and traces@mappings to core * Update traces-otel@mappings.yaml * Review feedback * Adapt `match` style in tests * Update docs/changelog/111091.yaml * Apply suggestions from code review Co-authored-by: Vishal Raj <vishal.raj@elastic.co> * Update x-pack/plugin/otel-data/src/main/resources/component-templates/traces-otel@mappings.yaml Co-authored-by: Carson Ip <carsonip@users.noreply.github.com> * Changing trace_flags to long Related discussion: elastic#111091 (comment) * Remove trace_flags see: elastic/opentelemetry-dev#368 (review) * Apply suggestions from code review Co-authored-by: Andrew Wilkins <axwalk@gmail.com> * Review feedback * Add store_array_source for span links * Define constant `data_stream.type` in `template.yaml`s * Create package-info.java * Move ecs-tsdb@mappings to index template Add test to verify that @Custom template can add dynamic templates with a higher precedence * Update metrics@mappings.json Remove summary_gauge and summary_counter since they are covered by summary_metrics * Move clusterService.getClusterSettings().addSettingsUpdateConsumer to registry * Fix code-style * Update x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs.tests.yml Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com> * Enable logsdb * Update traces@settings.json No lifecycle needed for OTel at this point --------- Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com> Co-authored-by: Carson Ip <carsonip@users.noreply.github.com> Co-authored-by: Vishal Raj <vishal.raj@elastic.co> Co-authored-by: Andrew Wilkins <axwalk@gmail.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Felix Barnsteiner <felix.barnsteiner@elastic.co>

@Custom

* Add YamlTemplateRegistry and OtelIndexTemplateRegistry with resource YAML files * Fix traces-otel template * Adding first yml tests * Base APMIndexTemplateRegistry on YamlTemplateRegistry * Update OTelPlugin.java * Update APMIndexTemplateRegistry.java * Update YamlIngestPipelineConfig.java * Adding traces tests * Update x-pack/plugin/otel-data/src/main/resources/component-templates/ecs-tsdb@mappings.yaml Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com> * Add mapper-version * Fix code-style * Rename `status.status_code` to `status.code` * Update otel@mappings.yaml Revert back to date due to missing support in ES|QL for date_nanos * Move dynamic_templates to metrics@mappings in core * Run gradlew :x-pack:plugin:core:spotlessApply * Update x-pack/plugin/otel-data/src/main/resources/component-templates/metrics-otel@mappings.yaml Co-authored-by: Carson Ip <carsonip@users.noreply.github.com> * Update 20_metic_tests.yml Workaround for TSDB timestamp issue: we push a custom template with higher priority and set time_series.start_time. * Update CODEOWNERS Adding obs-ds-intake-services as owner of the new otel-data plugin. Since we had some changes, also updating the owner of apm-data to the same team. * Change dynamic: strict to false * Skip "Reject invalid top level field" test * Update 20_metic_tests.yml * Add boolean as dimension test (skipping it for now) * Add booleans_to_keywords and enable corresponding test * Remove processor.event top level mapping Reason: for metrics and logs we can rely on the name of the datastream. For spans vs. transactions there are other fields we can use. * Remove booleans_to_keywords Because booleans are supported now as dimension on TSDB * Add alias service.language.name -> telemetry.sdk.language * cleanup * Update README.md * Update README.md * Update docs/changelog/111091.yaml * Move traces@settings and traces@mappings to core * Update traces-otel@mappings.yaml * Review feedback * Adapt `match` style in tests * Update docs/changelog/111091.yaml * Apply suggestions from code review Co-authored-by: Vishal Raj <vishal.raj@elastic.co> * Update x-pack/plugin/otel-data/src/main/resources/component-templates/traces-otel@mappings.yaml Co-authored-by: Carson Ip <carsonip@users.noreply.github.com> * Changing trace_flags to long Related discussion: elastic#111091 (comment) * Remove trace_flags see: elastic/opentelemetry-dev#368 (review) * Apply suggestions from code review Co-authored-by: Andrew Wilkins <axwalk@gmail.com> * Review feedback * Add store_array_source for span links * Define constant `data_stream.type` in `template.yaml`s * Create package-info.java * Move ecs-tsdb@mappings to index template Add test to verify that @Custom template can add dynamic templates with a higher precedence * Update metrics@mappings.json Remove summary_gauge and summary_counter since they are covered by summary_metrics * Move clusterService.getClusterSettings().addSettingsUpdateConsumer to registry * Fix code-style * Update x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs.tests.yml Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com> * Enable logsdb * Update traces@settings.json No lifecycle needed for OTel at this point --------- Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com> Co-authored-by: Carson Ip <carsonip@users.noreply.github.com> Co-authored-by: Vishal Raj <vishal.raj@elastic.co> Co-authored-by: Andrew Wilkins <axwalk@gmail.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Felix Barnsteiner <felix.barnsteiner@elastic.co>

elasticsearchmachine added v8.16.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels Jul 19, 2024

felixbarny reviewed Jul 22, 2024

View reviewed changes

x-pack/plugin/otel-data/src/main/resources/component-templates/ecs-tsdb@mappings.yaml Outdated Show resolved Hide resolved

gregkalapos commented Jul 22, 2024

View reviewed changes

x-pack/plugin/otel-data/src/main/resources/component-templates/traces-otel@mappings.yaml Outdated Show resolved Hide resolved

felixbarny reviewed Jul 23, 2024

View reviewed changes

x-pack/plugin/otel-data/src/main/resources/component-templates/otel@mappings.yaml Outdated Show resolved Hide resolved

felixbarny reviewed Jul 23, 2024

View reviewed changes

x-pack/plugin/otel-data/src/main/java/org/elasticsearch/xpack/oteldata/OTelPlugin.java Show resolved Hide resolved

felixbarny mentioned this pull request Jul 24, 2024

Add OpenTelemetry mappings #104455

Closed

felixbarny linked an issue Jul 24, 2024 that may be closed by this pull request

Add OpenTelemetry mappings #104731

Closed

felixbarny reviewed Jul 24, 2024

View reviewed changes

x-pack/plugin/otel-data/src/main/resources/component-templates/metrics-otel@mappings.yaml Outdated Show resolved Hide resolved

felixbarny reviewed Jul 25, 2024

View reviewed changes

x-pack/plugin/otel-data/src/main/resources/component-templates/otel@mappings.yaml Outdated Show resolved Hide resolved

strawgate reviewed Jul 25, 2024

View reviewed changes

carsonip reviewed Jul 26, 2024

View reviewed changes

x-pack/plugin/otel-data/src/main/resources/component-templates/metrics-otel@mappings.yaml Outdated Show resolved Hide resolved

felixbarny reviewed Jul 26, 2024

View reviewed changes

gregkalapos commented Jul 26, 2024

View reviewed changes

.github/CODEOWNERS Show resolved Hide resolved

gregkalapos commented Jul 26, 2024

View reviewed changes

x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_metic_tests.yml Outdated Show resolved Hide resolved

gregkalapos force-pushed the otel_plugin branch from 3bc869c to 6614c00 Compare July 30, 2024 13:43

carsonip reviewed Jul 31, 2024

View reviewed changes

x-pack/plugin/otel-data/src/main/resources/component-templates/otel@mappings.yaml Outdated Show resolved Hide resolved

gregkalapos force-pushed the otel_plugin branch from 5b4b65f to 9c4ea86 Compare July 31, 2024 14:09

gregkalapos and others added 10 commits August 5, 2024 19:18

Add YamlTemplateRegistry and OtelIndexTemplateRegistry with resource …

96c5c22

…YAML files

Fix traces-otel template

d5861bb

Adding first yml tests

e58d359

Base APMIndexTemplateRegistry on YamlTemplateRegistry

4b7414c

Update OTelPlugin.java

b65bb90

Update APMIndexTemplateRegistry.java

fc82339

Update YamlIngestPipelineConfig.java

72a6769

Adding traces tests

9cd50ee

Update x-pack/plugin/otel-data/src/main/resources/component-templates…

953968f

…/ecs-tsdb@mappings.yaml Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com>

Add mapper-version

6206ab3

axw approved these changes Aug 13, 2024

View reviewed changes

x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs.tests.yml Outdated Show resolved Hide resolved

felixbarny and others added 4 commits August 13, 2024 11:47

Move ecs-tsdb@mappings to index template

16234d7

Add test to verify that @Custom template can add dynamic templates with a higher precedence

Update metrics@mappings.json

0389c37

Remove summary_gauge and summary_counter since they are covered by summary_metrics

Move clusterService.getClusterSettings().addSettingsUpdateConsumer to…

9b93b75

… registry

Fix code-style

3152f99

felixbarny reviewed Aug 14, 2024

View reviewed changes

x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-spec/test/20_logs.tests.yml Outdated Show resolved Hide resolved

felixbarny approved these changes Aug 14, 2024

View reviewed changes

Update x-pack/plugin/otel-data/src/yamlRestTest/resources/rest-api-sp…

680e519

…ec/test/20_logs.tests.yml Co-authored-by: Felix Barnsteiner <felixbarny@users.noreply.github.com>

gregkalapos mentioned this pull request Aug 14, 2024

x-pack-otel plugin follow-up tasks #111877

Open

5 tasks

Enable logsdb

ef3468f

gregkalapos requested a review from carsonip August 14, 2024 10:47

carsonip approved these changes Aug 14, 2024

View reviewed changes

gregkalapos requested a review from dakrone August 14, 2024 11:25

carsonip reviewed Aug 14, 2024

View reviewed changes

x-pack/plugin/core/template-resources/src/main/resources/traces@settings.json Outdated Show resolved Hide resolved

Update traces@settings.json

3bb6737

No lifecycle needed for OTel at this point

dakrone self-assigned this Aug 15, 2024

dakrone approved these changes Aug 15, 2024

View reviewed changes

Merge branch 'main' into otel_plugin

9f18034

gregkalapos merged commit 4e1a84c into elastic:main Aug 19, 2024
15 checks passed

gregkalapos deleted the otel_plugin branch August 19, 2024 07:24

gregkalapos mentioned this pull request Sep 23, 2024

Remove dynamic templates from otel-plugin that set index:false #113409

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x-pack/plugin/otel: introduce x-pack-otel plugin #111091

x-pack/plugin/otel: introduce x-pack-otel plugin #111091

gregkalapos commented Jul 19, 2024 •

edited

Loading

strawgate Jul 25, 2024

gregkalapos Jul 25, 2024

strawgate Aug 13, 2024 •

edited

Loading

gregkalapos Aug 19, 2024

felixbarny commented Jul 30, 2024

elasticmachine commented Jul 30, 2024

gregkalapos commented Aug 12, 2024

axw left a comment

axw commented Aug 14, 2024

felixbarny commented Aug 14, 2024

axw commented Aug 14, 2024

gregkalapos commented Aug 14, 2024

dakrone left a comment

gregkalapos commented Aug 19, 2024

x-pack/plugin/otel: introduce x-pack-otel plugin #111091

x-pack/plugin/otel: introduce x-pack-otel plugin #111091

Conversation

gregkalapos commented Jul 19, 2024 • edited Loading

strawgate Jul 25, 2024

Choose a reason for hiding this comment

gregkalapos Jul 25, 2024

Choose a reason for hiding this comment

strawgate Aug 13, 2024 • edited Loading

Choose a reason for hiding this comment

gregkalapos Aug 19, 2024

Choose a reason for hiding this comment

felixbarny commented Jul 30, 2024

elasticmachine commented Jul 30, 2024

gregkalapos commented Aug 12, 2024

axw left a comment

Choose a reason for hiding this comment

axw commented Aug 14, 2024

felixbarny commented Aug 14, 2024

axw commented Aug 14, 2024

gregkalapos commented Aug 14, 2024

dakrone left a comment

Choose a reason for hiding this comment

gregkalapos commented Aug 19, 2024

gregkalapos commented Jul 19, 2024 •

edited

Loading

strawgate Aug 13, 2024 •

edited

Loading