Corrupt msgpack encoding #970
Comments
|
This is certainly quite odd. For reference, we encode MessagePack manually, rather than using an off-the-shelf library. A few questions:
|
|
Turns out this is an issue with moleculer (microservice framework) implementation of a Datadog trace exporter. (See moleculerjs/moleculer#725 (comment).) Basically, it's overwriting the Additionally, once a span with a bad id has been encoded, all future encodes will also be corrupt, even spans with valid ids (like the one in the original post's example), due to the reuse of a previously-corrupted I'm not sure if there's any work to be done here, given that the issue appears to be misuse of the library's private internals, but this may spark some ideas for improvement on your end. I will also likely file the agent crash as a memory leak issue. |
|
Using these externally is definitely not supported since they are internal APIs. I left a comment on moleculerjs/moleculer#725 to try to see how we can improve the situation. |
|
#976 truncates the IDs to 8 bytes. This should prevent corrupting the |
|
Fix released in 0.21.1 |
Describe the bug
I noticed my agents are periodically crashing after a bunch of trace decoding errors.
Agent log
So I turned on debug logging in
dd-tracein my app to inspect the trace payloads. Here's a sample span that was logged by the tracer:The printed buffer appears to be corrupt:
It should be:
91 = 1-item array 8c = 12-entry map -- a8 = 8-byte string 74 72 61 63 65 5f 69 64 = "trace_id" cf = uint64 31 73 1c 1f ac 15 ce ea -- - b9 6e 28 38 97 e4 3d 34 + a7 = 7-byte string + 73 70 61 6e 5f 69 64 = "span_id" cf = uint64 31 73 1c 1f ac 15 ce ea -- - 98 b4 51 b9 6c b6 c0 16 13 6b b4 d0 ee 4d 00 + a5 = 5-byte string + 73 74 61 72 74 = "start" + d3 = int64 + 16 13 6b b4 d0 ee 4d 18 = 1590733517735480600 -- a8 = 8-byte string 64 75 72 61 74 69 6f 6e = "duration" d3 = int64 00 00 00 00 00 19 7f e7 = 1671143 -- a5 = 5-byte string 65 72 72 6f 72 = "error" 00 = 0 -- a9 = 9-byte string 70 61 72 65 6e 74 5f 69 64 = "parent_id" cf = uint64 00 00 00 00 00 00 00 00 -- a4 = 4-byte string 6e 61 6d 65 = "name" a8 = 8-byte string 70 67 2e 71 75 65 72 79 = "pg.query" -- [...]Oddly, when I add this span data to the encoder unit tests in the
dd-traceproject, it encodes correctly. So I'm stumped. Any idea why the printed trace and encoded buffer don't match? Is this the cause of the agent decoding errors?Environment
The text was updated successfully, but these errors were encountered: