[exporter/elasticsearch] OTel mode @timestamp precision causes TSDB version conflict

[carsonip]

Component(s)

exporter/elasticsearch

What happened?

Description

When sending high traffic of metrics to Elasticsearch 8.16 using OTel mapping mode, it is possible to get error logs.

2024-09-18T15:14:23.115+0100	error	elasticsearchexporter@v0.109.0/bulkindexer.go:332	failed to index document	{"kind": "exporter", "data_type": "metrics", "name": "elasticsearch", "index": ".ds-metrics-generic.otel-default-2024.09.18-000001", "error.type": "version_conflict_engine_exception", "error.reason": "[nTN_4QYSjpM0biAuAAABkgV71wQ][QFI9z_AVWfXPjRal9mmHxBaEHFGlYZitCG-SF5Dv4af6d4idRLDI76NjRaBnhiIN2lxixVbyjo-FtJL-g-Mh1hs@2024-09-18T14:14:15.044Z]: version conflict, document already exists (current version [1])"}

It affects OTel mapping mode because the built-in otel-data plugin in ES will use TSDB and date @timestamp. In general, it affects cases where metrics is sent to TSDB where mapping @timestamp not of type date_nanos.

This is due to ES 8.16 grouping metrics in milliseconds as opposed to nanoseconds in elasticsearchexporter. This is a misalignment between ES and elasticsearchexporter, but the fix should go into ES.

Steps to Reproduce

In ES dev tools,

POST metrics-bar.otel-default/_doc
{"@timestamp":"2024-09-18T14:14:06.988158400Z","data_stream":{"dataset":"bar.otel","namespace":"default","type":"metrics"},"metrics":{"process.runtime.dotnet.jit.methods_compiled.count":17380},"resource":{"attributes":{"container.id":"b5028825d0c55c0811c3a6c9be8cc50db8d8d3975501778e28203288a9c5a6c7","docker.cli.cobra.command_path":"docker%20compose","host.name":"b5028825d0c5","service.name":"cartservice","telemetry.sdk.language":"dotnet","telemetry.sdk.name":"opentelemetry","telemetry.sdk.version":"1.9.0"},"dropped_attributes_count":0},"scope":{"dropped_attributes_count":0,"name":"OpenTelemetry.Instrumentation.Runtime","version":"1.9.0"},"start_timestamp":"2024-09-18T11:02:07.006501300Z"}

POST metrics-bar.otel-default/_doc
{"@timestamp":"2024-09-18T14:14:06.988159300Z","data_stream":{"dataset":"bar.otel","namespace":"default","type":"metrics"},"metrics":{"process.runtime.dotnet.thread_pool.queue.length":0},"resource":{"attributes":{"container.id":"b5028825d0c55c0811c3a6c9be8cc50db8d8d3975501778e28203288a9c5a6c7","docker.cli.cobra.command_path":"docker%20compose","host.name":"b5028825d0c5","service.name":"cartservice","telemetry.sdk.language":"dotnet","telemetry.sdk.name":"opentelemetry","telemetry.sdk.version":"1.9.0"},"dropped_attributes_count":0},"scope":{"dropped_attributes_count":0,"name":"OpenTelemetry.Instrumentation.Runtime","version":"1.9.0"},"start_timestamp":"2024-09-18T11:02:07.006594400Z"}

Expected Result

No error

Actual Result

2nd metric not indexed

Collector version

v0.109.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @carsonip @lahsivjar

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[carsonip]

/label -needs-triage