Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: logger metadata can mangle span_ids #117

Closed

Conversation

gerbal
Copy link
Contributor

@gerbal gerbal commented Sep 9, 2020

We are deploying Spandex to our elixir services and have discovered in some circumstances it the Elixir Logger can lose precision on very large integers. This has resulted in oddly truncated trace and span ids in our logs like 6123256812116107000. The underlying cause is unclear, but the whole issue can be avoided if the trace_id and span_id are stored as strings in logger metadata.

Is this a useful change for anyone else?

We have seen several cases were some component of Elixir Logger can lose precision on very large integers stored in log metadata. This fix casts the trace_id to string to avoid this issue.
@zachdaniel
Copy link
Member

@gerbal so the trace/span ids are in datadog w/o the trailing zeroes? I suspect its related to this https://github.com/spandex-project/spandex_datadog/blob/3344868ac5eca8f7ce6d9558d970edfd0d109073/lib/spandex_datadog/adapter.ex#L24

@gerbal
Copy link
Contributor Author

gerbal commented Sep 9, 2020

The trace_id is actually 6123256812116107452 and what's logged is 6123256812116107000

@zachdaniel
Copy link
Member

Yeah, that is really strange...I'd really suggest seeing if you can reproduce the specific issue of integers losing precision when being transferred to logger metadata. That would be an issue w/ Elixir/logger I imagine

@GregMefford
Copy link
Member

I didn’t have a chance to try to repro this yet, but if you’re able to give some more details about what versions of Elixir and OTP you’re seeing this on, that would be helpful. I know there are some recent changes to the way Logger works and others may have just not updated/noticed yet if there are issues there. ❤️ 🚀

@zachdaniel
Copy link
Member

Yeah, I tried to reproduce it but I couldn't.

@aselder
Copy link
Contributor

aselder commented Dec 3, 2020

The PR I just submitted is a dup of this. I can confirm the behavior, see #126 for details.

@aselder
Copy link
Contributor

aselder commented Dec 3, 2020

For clarity, the issues is with log ingesters (namely datadog), treating JSON integers as JS Numbers (aka float64) and losing precision.

@zachdaniel zachdaniel closed this Dec 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants