Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transform logs to spans (PR WIP - #3003) #3071

Closed
gramidt opened this issue Apr 12, 2021 · 1 comment
Closed

Transform logs to spans (PR WIP - #3003) #3071

gramidt opened this issue Apr 12, 2021 · 1 comment
Labels
data:logs Logs related issues data:traces Trace related issues

Comments

@gramidt
Copy link
Member

gramidt commented Apr 12, 2021

Is your feature request related to a problem? Please describe.
Some systems such as load balancers, proxies, or IoT devices may not have native trace support at this time, but have the capabilities to generate the necessary data to populate a span. These systems also may not allow importing external libraries, so directly adding in an OTel library may not be an option, such as in my use case.

Example: Imagine you have a proxy that has a robust scripting engine where you can manipulate inbound requests with code and this proxy doesn't currently natively support W3C Trace Context. You create the script to support W3C Trace Context and upon the request response, you're able to log out the necessary data to populate a span.

Describe the solution you'd like
A "log-to-span" exporter that is able to convert an OTel Log to an OTel Span. This exporter should heavily rely on the upstream Log Receiver to parse fields into Attributes on the Log. With this exporter, the following is an example of what can be achieved.

  1. log that contains all fields for a span ->
  2. log receiver (parses logs and outputs logs with attributes) ->
  3. [processors] ->
  4. logtospanexporter (receives log with span fields as attributes, transform the log into a span and exports via OTLP over gRPC) ->
  5. downstream OTel Collector receives OTLP spans and processes them via the configured tracing pipelines

Describe alternatives you've considered

  • Add custom log receiver that transform logs to spans (this approach would limit how logs are received and would require duplication of key parsing methods)
  • Start working on the "Signal Translation Proposal" (Signal Translation Proprosal opentelemetry-collector#2336). This is in my opinion is the BEST mid-to-long-term option; however, this functionality is absolutely critical to have now and cannot wait until we're able to solidify the implementation of this proposal. Once this proposal is finalized, this exporter could be deprecated in favor of a log to trace translator, where some of the "logtotracexporter" code could likely be reused.

Additional context
An initial PoC was created in a private distribution and I have already started a more robust/configurable version of this exporter in PR #3003

Slack thread within the #otel-collector where this was initially discussed: https://cloud-native.slack.com/archives/C01N6P7KR6W/p1616472939047100

@gramidt
Copy link
Member Author

gramidt commented Apr 12, 2021

@bogdandrutu @tigrannajaryan - Please assign to me and add labels.

alexperez52 referenced this issue in open-o11y/opentelemetry-collector-contrib Aug 18, 2021
Fix the broken build due to the new storage.Appender and new configuration loading API.

Fixes #3071.
Closes #3066.
@alolita alolita added data:logs Logs related issues data:traces Trace related issues labels Sep 30, 2021
@gramidt gramidt closed this as completed Jul 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data:logs Logs related issues data:traces Trace related issues
Projects
None yet
Development

No branches or pull requests

2 participants