Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use OTLP headers to dynamically set prometheusremotewrite exporter values #30310

Closed
justindesilets opened this issue Jan 5, 2024 · 4 comments
Labels
enhancement New feature or request exporter/prometheusremotewrite needs triage New item requiring triage

Comments

@justindesilets
Copy link

Component(s)

exporter/prometheusremotewrite

Is your feature request related to a problem? Please describe.

When the otel-collector is started, the prometheusremotewrite reads in the configuration and then runs statically for the duration of the process. This prevents you from building a horizontal ingestion point where you could have multiple tenants sending metrics in and then having them dispatched to different endpoints / tenant IDs.

Describe the solution you'd like

Using the OTEL_EXPORTER_OTLP_METRICS_HEADERS with each tenant's application, they would send their remote URL and credentials as a headers. The prometheusremoteagent would then apply this to its writes to the matching Cortex backend, storing each tenants metrics into their own tenant space.

Describe alternatives you've considered

No response

Additional context

The intent here is to be able to provide an scalable multi-tenant ingestion point where internal teams could send all of their metrics to. This would allow for a self-service model for these tenants to easily be able to silo their data from each other when storing it into Cortex.

@justindesilets justindesilets added enhancement New feature or request needs triage New item requiring triage labels Jan 5, 2024
Copy link
Contributor

github-actions bot commented Jan 5, 2024

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

Copy link
Contributor

github-actions bot commented Mar 6, 2024

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@jmichalek132
Copy link
Contributor

I am not sure this is possible to implement, as far as I know the the context of the headers of incoming metrics into OTEL collector is not available to exporters. but maybe similar to https://github.com/blind-oracle/cortex-tenant we could use a configurable resource attribute as tenant.

@jmichalek132
Copy link
Contributor

A discussed during the Opentelemetry Prometheus WG the goal of this exporter is to mirror prometheus remote write capabilities. Since Prometheus doesn't support this, we don't want to diverge from prometheus, also the use case you are describing should be solvable while using existing proxies such as https://cortexmetrics.io/docs/guides/auth/#cortex-tenant or https://github.com/cortexproject/auth-gateway.
Could you @dashpole please close this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request exporter/prometheusremotewrite needs triage New item requiring triage
Projects
None yet
Development

No branches or pull requests

3 participants