Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Breakdown metrics warning after running for a while #383

Closed
SylvainJuge opened this issue Aug 21, 2024 · 0 comments · Fixed by #384
Closed

Breakdown metrics warning after running for a while #383

SylvainJuge opened this issue Aug 21, 2024 · 0 comments · Fixed by #384

Comments

@SylvainJuge
Copy link
Member

SylvainJuge commented Aug 21, 2024

Describe the bug

After running for a while, the agent continuously spam the standard output with this warning from the otel SDK:

WARN io.opentelemetry.sdk.metrics.internal.state.DefaultSynchronousMetricStorage - Instrument elastic.span_breakdown has exceeded the maximum allowed cardinality (1999)

The reason we have a high cardinality here is because we capture the span self time as a metric attribute to allow using it in ingest pipeline later.

Given the current state of breakdown metrics, we should probably:

  • make the feature opt-in through configuration, which requires to migrate it first to autoconfig
  • unplug it for now, but keep the code around for later use.
  • remove the feature completely until its ready for prime-time
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant