Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DI] Workaround bug in AsyncLocalStorage which would otherwise throw #5290

Merged
merged 4 commits into from
Feb 19, 2025

Conversation

watson
Copy link
Collaborator

@watson watson commented Feb 18, 2025

What does this PR do?

This is a workaround for, what seems like a bug in Node.js core, that seems to trigger when, among other things, a lot of timers are being created very rapidly. This makes the call to setTimeout throw an error from within AsyncLocalStorage._propagate with the following TypeError:

Cannot read properties of undefined (reading 'Symbol(kResourceStore)')

Source of the bug: https://github.com/nodejs/node/blob/v18.20.6/lib/async_hooks.js#L312

Motivation

Don't crash customer applications.

Plugin Checklist

Additional Notes

Copy link
Collaborator Author

watson commented Feb 18, 2025

This stack of pull requests is managed by Graphite. Learn more about stacking.

Copy link

github-actions bot commented Feb 18, 2025

Overall package size

Self size: 8.75 MB
Deduped: 94.95 MB
No deduping: 95.47 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.4.0 | 29.44 MB | 29.44 MB | | @datadog/native-appsec | 8.4.0 | 19.25 MB | 19.26 MB | | @datadog/native-iast-taint-tracking | 3.3.0 | 13.77 MB | 13.78 MB | | @datadog/pprof | 5.5.1 | 9.79 MB | 10.17 MB | | protobufjs | 7.2.5 | 2.77 MB | 5.16 MB | | @datadog/native-iast-rewriter | 2.8.0 | 2.6 MB | 2.74 MB | | @opentelemetry/core | 1.14.0 | 872.87 kB | 1.47 MB | | @datadog/native-metrics | 3.1.0 | 1.06 MB | 1.46 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | import-in-the-middle | 1.11.2 | 112.74 kB | 835.4 kB | | source-map | 0.7.4 | 226 kB | 226 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | lru-cache | 7.18.3 | 133.92 kB | 133.92 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.0 | 109.9 kB | 109.9 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 5.3.2 | 53.63 kB | 53.63 kB | | shell-quote | 1.8.1 | 44.96 kB | 44.96 kB | | istanbul-lib-coverage | 3.2.0 | 29.34 kB | 29.34 kB | | rfdc | 1.3.1 | 25.21 kB | 25.21 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | dc-polyfill | 0.1.4 | 23.1 kB | 23.1 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | semifies | 1.0.0 | 15.84 kB | 15.84 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.3 | 4.47 kB | 4.47 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

Copy link

codecov bot commented Feb 18, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.90%. Comparing base (172bc66) to head (22ab9c4).
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5290      +/-   ##
==========================================
+ Coverage   80.80%   80.90%   +0.09%     
==========================================
  Files         487      487              
  Lines       21710    21797      +87     
==========================================
+ Hits        17542    17634      +92     
+ Misses       4168     4163       -5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented Feb 18, 2025

Datadog Report

Branch report: watson/DEBUG-3526/nodejs-core-bug-workaround
Commit report: bd7cd0a
Test service: dd-trace-js-integration-tests

✅ 0 Failed, 674 Passed, 0 Skipped, 16m 8.67s Total Time

@pr-commenter
Copy link

pr-commenter bot commented Feb 18, 2025

Benchmarks

Benchmark execution time: 2025-02-19 09:23:56

Comparing candidate commit 22ab9c4 in PR branch watson/DEBUG-3526/nodejs-core-bug-workaround with baseline commit 172bc66 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 915 metrics, 18 unstable metrics.

Copy link
Contributor

@cbeauchesne cbeauchesne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about adding failing system-tests scenarios?

it's here : https://github.com/DataDog/dd-trace-js/blob/master/.github/workflows/system-tests.yml#L34

Just add a line : scenarios: DEBUGGER_PII_REDACTION, DEBUGGER_PROBES_SNAPSHOT

@watson watson force-pushed the watson/DEBUG-3526/nodejs-core-bug-workaround branch from 70ceee5 to 4d83e30 Compare February 18, 2025 15:13
@watson watson requested a review from a team as a code owner February 18, 2025 15:13
This is a workaround for, what seems like a bug in Node.js core, that
seems to trigger when, among other things, a lot of timers are being
created very rapidly. This makes the call to `setTimeout` throw an error
from within `AsyncLocalStorage._propagate` with the following TypeError:

    Cannot read properties of undefined (reading 'Symbol(kResourceStore)')
@watson watson force-pushed the watson/DEBUG-3526/nodejs-core-bug-workaround branch from 4d83e30 to 75e555e Compare February 19, 2025 09:00
@watson watson requested a review from cbeauchesne February 19, 2025 09:00
cbeauchesne
cbeauchesne previously approved these changes Feb 19, 2025
@watson watson enabled auto-merge (squash) February 19, 2025 10:14
@watson watson merged commit 77abd1e into master Feb 19, 2025
386 of 388 checks passed
@watson watson deleted the watson/DEBUG-3526/nodejs-core-bug-workaround branch February 19, 2025 10:17
watson added a commit that referenced this pull request Feb 19, 2025
…5290)

This is a workaround for, what seems like a bug in Node.js core, that
seems to trigger when, among other things, a lot of timers are being
created very rapidly. This makes the call to `setTimeout` throw an error
from within `AsyncLocalStorage._propagate` with the following TypeError:

    Cannot read properties of undefined (reading 'Symbol(kResourceStore)')
@watson watson mentioned this pull request Feb 19, 2025
watson added a commit that referenced this pull request Feb 20, 2025
…5290)

This is a workaround for, what seems like a bug in Node.js core, that
seems to trigger when, among other things, a lot of timers are being
created very rapidly. This makes the call to `setTimeout` throw an error
from within `AsyncLocalStorage._propagate` with the following TypeError:

    Cannot read properties of undefined (reading 'Symbol(kResourceStore)')
@watson watson mentioned this pull request Feb 20, 2025
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants