Inject trace info as comment to MongoDB operation when dbm propagation is enabled. #5230

lu-zhengda · 2025-02-07T20:13:04Z

What does this PR do?

This PR introduces DBM propagation mode support for MongoDB operations, aligning with the existing functionality in relational database such as PostgreSQL and MySQL. DBM propagation enables linking APM traces with DBM samples through trace comment injection, providing enhanced observability into database operations.

Key Features:

Trace Propagation via Comments:
The tracer now supports injecting trace metadata into MongoDB commands using the comment field.
Configurable Propagation Mode:
Users can configure the behavior via the dbmPropagationMode setting:
- disabled – No trace metadata is injected.
- service – Injects only service-level information.
- full – Injects both service and span-level information.

Example Usage:

Set dbmPropagationMode in tracer config

// Load the dd-trace library for tracing
const tracer = require("dd-trace").init({
  service: "express-mongodb-app", // Your service name
  env: "dev", // Your environment
  ......  // Your other tracer configs
  dbmPropagationMode: "full",
});

With DBM propagation enabled, MongoDB commands will include a trace comment similar to:

Motivation

Improved Traceability:
With trace information attached, MongoDB operation samples in DBM can now be linked to APM traces, allowing users to analyze query performance in the context of distributed tracing.
https://datadoghq.atlassian.net/browse/DBMON-4990

Plugin Checklist

Additional Notes

github-actions · 2025-02-07T20:13:58Z

Overall package size

Self size: 8.69 MB
Deduped: 94.89 MB
No deduping: 95.41 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.4.0 | 29.44 MB | 29.44 MB | | @datadog/native-appsec | 8.4.0 | 19.25 MB | 19.26 MB | | @datadog/native-iast-taint-tracking | 3.3.0 | 13.77 MB | 13.78 MB | | @datadog/pprof | 5.5.1 | 9.79 MB | 10.17 MB | | protobufjs | 7.2.5 | 2.77 MB | 5.16 MB | | @datadog/native-iast-rewriter | 2.8.0 | 2.6 MB | 2.74 MB | | @opentelemetry/core | 1.14.0 | 872.87 kB | 1.47 MB | | @datadog/native-metrics | 3.1.0 | 1.06 MB | 1.46 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | import-in-the-middle | 1.11.2 | 112.74 kB | 835.4 kB | | source-map | 0.7.4 | 226 kB | 226 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | lru-cache | 7.18.3 | 133.92 kB | 133.92 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.0 | 109.9 kB | 109.9 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 5.3.2 | 53.63 kB | 53.63 kB | | shell-quote | 1.8.1 | 44.96 kB | 44.96 kB | | istanbul-lib-coverage | 3.2.0 | 29.34 kB | 29.34 kB | | rfdc | 1.3.1 | 25.21 kB | 25.21 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | dc-polyfill | 0.1.4 | 23.1 kB | 23.1 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | semifies | 1.0.0 | 15.84 kB | 15.84 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.3 | 4.47 kB | 4.47 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

codecov · 2025-02-07T20:14:34Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.25%. Comparing base (b8c03bd) to head (470722f).
Report is 8 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5230      +/-   ##
==========================================
+ Coverage   81.09%   81.25%   +0.15%     
==========================================
  Files         480      487       +7     
  Lines       21438    21716     +278     
==========================================
+ Hits        17386    17646     +260     
- Misses       4052     4070      +18

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

datadog-datadog-prod-us1 · 2025-02-07T20:18:55Z

Datadog Report

Branch report: zhengda.lu/mongo-dbm-trace
Commit report: 105bac7
Test service: dd-trace-js-integration-tests

✅ 0 Failed, 628 Passed, 0 Skipped, 14m 56.72s Total Time

pr-commenter · 2025-02-10T20:23:54Z

Benchmarks

Benchmark execution time: 2025-02-12 22:44:05

Comparing candidate commit 470722f in PR branch zhengda.lu/mongo-dbm-trace with baseline commit b8c03bd in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 907 metrics, 26 unstable metrics.

packages/dd-trace/src/plugins/database.js

Co-authored-by: Bryan English <bryan.english@datadoghq.com>

wconti27

The tests and plugin change looks good to me. But we should put the injection functionality within the same function as the existing DBM propagation.

packages/dd-trace/src/plugins/database.js

wconti27 · 2025-02-11T19:46:47Z

packages/dd-trace/src/plugins/database.js

+    if (command.comment) {
+      // if the command already has a comment, append the dbm trace comment
+      if (typeof command.comment === 'string') {
+        command.comment += `,${dbmTraceComment}`


Just because of curiosity, why are we appending the DBM comment to the end of the command? As opposed to previous MySQL injection where we prepend it to the beginning of the comment.

This is due the how MongoDB handles database operations as compared to relational database like mysql. MongoDB database commands are sent as object instead of SQL string. In mysql we chose to prepend comment to the beginning of the SQL statement because we want to avoid appending comment after optional trailing ;. We don't have such edge cases for MongoDB because comment is an attribute in the command object. I chose append so that the tracer comment appears after user comment.

Can you add a test that ensures we still work even when the user provides a comment? It looks like the comments can be a string or an array. Can you create a test for both situations? It should be sufficient to add this to only one of the two test files.

Is whatever DBM doing on the backend compatible with both string and array approaches? I'm sure it is but just want to make sure.

Also, with the comment concatenation approach is there any user-supplied comment pattern that can break parsing of the DBM comment? E.g. if the user sets a comment to foo= or ,,, or asdf"?

Does the backend only consider the first or last occurrence of DBM comment fields? If it's just the first then the comment should be prepended, and if it's the last it should be appended like you're doing here. E.g. if the user comment is ddpv=fooboar does the real ddpv=1.2.3 appended by the tracer still work as expected?

Test cases updated to cover command with str or array comments.
Regarding prepend vs. append, DBM extracts tags from raw comments. The last key:value pair wins when it comes to duplicate keys.

with the comment concatenation approach is there any user-supplied comment pattern that can break parsing of the DBM comment? E.g. if the user sets a comment to foo= or ,,, or asdf"

Any user supplied comments that do not follow the key='value' style will be safely ignored. DBM backend extracts the key='value' pair from the comment and leaves the comment untouched.

Is whatever DBM doing on the backend compatible with both string and array approaches?

DBM backend unmarshals both string and array into an array.

packages/dd-trace/src/plugins/database.js

packages/datadog-plugin-mongodb-core/src/index.js

tlhunter

Left some comments requesting changes

lu-zhengda · 2025-02-12T22:30:13Z

docker-compose.yml

@@ -37,7 +37,7 @@ services:
    ports:
      - "127.0.0.1:6379:6379"
  mongo:
-    image: circleci/mongo:3.6
+    image: circleci/mongo:4.4


bump mongo image to 4.4 because prior to 4.4 only str comments are allowed. newer mongo version allows us to test array comments.

inject dbm trace comment

abfb239

add service mode test

2f887c5

lu-zhengda added the semver-minor label Feb 7, 2025

lu-zhengda added 2 commits February 10, 2025 20:12

add unit test to verify both full and service mode

f84e138

update test

a2e6c25

lu-zhengda added 7 commits February 10, 2025 20:32

fix lint

cab568b

use find query

f14dd0f

fix timeout

d4048c4

remove done

8c06bdc

add tests to mongodb-core

a129ed3

fix service mode full

2bc6ccb

remove custom timeout

1494f45

lu-zhengda marked this pull request as ready for review February 10, 2025 22:51

lu-zhengda requested review from a team as code owners February 10, 2025 22:51

lu-zhengda requested a review from a team February 10, 2025 22:51

dalextorres previously approved these changes Feb 11, 2025

View reviewed changes

tlhunter added the integration-mongodb label Feb 11, 2025

bengl requested changes Feb 11, 2025

View reviewed changes

packages/dd-trace/src/plugins/database.js Outdated Show resolved Hide resolved

Update packages/dd-trace/src/plugins/database.js

fecc4f2

Co-authored-by: Bryan English <bryan.english@datadoghq.com>

lu-zhengda dismissed dalextorres’s stale review via fecc4f2 February 11, 2025 19:38

wconti27 requested changes Feb 11, 2025

View reviewed changes

merge duplicate code

145b3bc

lu-zhengda requested review from a team as code owners February 11, 2025 20:37

lu-zhengda requested a review from nikita-tkachenko-datadog February 11, 2025 20:37

lu-zhengda force-pushed the zhengda.lu/mongo-dbm-trace branch from e3015f5 to 145b3bc Compare February 11, 2025 20:43

lu-zhengda removed request for a team and nikita-tkachenko-datadog February 11, 2025 20:44

Merge branch 'master' into zhengda.lu/mongo-dbm-trace

c6547ac

lu-zhengda requested review from wconti27 and bengl February 11, 2025 21:01

tlhunter reviewed Feb 12, 2025

View reviewed changes

packages/dd-trace/src/plugins/database.js Show resolved Hide resolved

tlhunter reviewed Feb 12, 2025

View reviewed changes

packages/datadog-plugin-mongodb-core/src/index.js Show resolved Hide resolved

tlhunter requested changes Feb 12, 2025

View reviewed changes

lu-zhengda added 2 commits February 12, 2025 22:04

add tests

e224aae

add tests to verify command with comments

7d5b6db

lu-zhengda commented Feb 12, 2025

View reviewed changes

fix lint

470722f

lu-zhengda requested review from tlhunter and dalextorres February 12, 2025 22:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Inject trace info as comment to MongoDB operation when dbm propagation is enabled. #5230

Inject trace info as comment to MongoDB operation when dbm propagation is enabled. #5230

lu-zhengda commented Feb 7, 2025 •

edited

Loading

github-actions bot commented Feb 7, 2025 •

edited

Loading

codecov bot commented Feb 7, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Feb 7, 2025 •

edited

Loading

pr-commenter bot commented Feb 10, 2025 •

edited

Loading

wconti27 left a comment

wconti27 Feb 11, 2025

lu-zhengda Feb 11, 2025

tlhunter Feb 12, 2025

lu-zhengda Feb 12, 2025

lu-zhengda Feb 12, 2025 •

edited

Loading

lu-zhengda Feb 12, 2025 •

edited

Loading

tlhunter left a comment

lu-zhengda Feb 12, 2025

Inject trace info as comment to MongoDB operation when dbm propagation is enabled. #5230

Are you sure you want to change the base?

Inject trace info as comment to MongoDB operation when dbm propagation is enabled. #5230

Conversation

lu-zhengda commented Feb 7, 2025 • edited Loading

What does this PR do?

Motivation

Plugin Checklist

Additional Notes

github-actions bot commented Feb 7, 2025 • edited Loading

Overall package size

codecov bot commented Feb 7, 2025 • edited Loading

Codecov Report

datadog-datadog-prod-us1 bot commented Feb 7, 2025 • edited Loading

Datadog Report

pr-commenter bot commented Feb 10, 2025 • edited Loading

Benchmarks

wconti27 left a comment

Choose a reason for hiding this comment

wconti27 Feb 11, 2025

Choose a reason for hiding this comment

lu-zhengda Feb 11, 2025

Choose a reason for hiding this comment

tlhunter Feb 12, 2025

Choose a reason for hiding this comment

lu-zhengda Feb 12, 2025

Choose a reason for hiding this comment

lu-zhengda Feb 12, 2025 • edited Loading

Choose a reason for hiding this comment

lu-zhengda Feb 12, 2025 • edited Loading

Choose a reason for hiding this comment

tlhunter left a comment

Choose a reason for hiding this comment

lu-zhengda Feb 12, 2025

Choose a reason for hiding this comment

lu-zhengda commented Feb 7, 2025 •

edited

Loading

github-actions bot commented Feb 7, 2025 •

edited

Loading

codecov bot commented Feb 7, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Feb 7, 2025 •

edited

Loading

pr-commenter bot commented Feb 10, 2025 •

edited

Loading

lu-zhengda Feb 12, 2025 •

edited

Loading

lu-zhengda Feb 12, 2025 •

edited

Loading