V4.48.0 proposal #4787
Merged
V4.48.0 proposal #4787
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… info from GHA (#4745)
Take a "snapshot" of the variables that are in scope when a probe is hit (except the global scope, which intentionally have been omitted since it's too noisy): - For each variable in scope, we traverse objects and their properties up to `maxReferenceDepth` deep (default is 3 levels). - Strings are truncated to `maxLength` (default is 255 characters). - Binary data is converted to strings with appropriate escaping of non printable characters (the `maxLength` limit is also applied) Out of scope: - Information about `this` is not captured. - maxCollectionSize limit - maxFieldCount limit - Special handling for snapshots larger than 1MB (e.g. snapshot pruning or something simpler) - PII redaction
* add protobufjs schemas support for DSM
* add avro (avsc) schemas support for DSM
* use AsyncLocalStorage instead of our home-grown solutions The comment in the file that selected a storage implementation suggested just using AsyncLocalStorage once it supports triggerAsyncResource(). That said, literally zero of our code uses triggerAsyncResource(), so this is assumed to be historical and no longer relevant. Switching to stock AsyncLocalStorage will enable the usage of TracingChannel in the future. * self-contain profiling's AsyncLocalStorage channel usage * remove flag detection
* Upgrade iast rewriter version to 2.5.0 * Implement tplOperator tracking method
* rasp lfi and iast using rasp fs-plugin * Add rasp lfi capability in RC * Handle aborted operations in fs instrumentation * enable test without express * cleanup and console log to debug test error * Do not throw * another test * Try increasing timeout * Enable debug again * Enable debug again * increase timeout a lot * increase timeout more * New lfi test * Increase test timeout * print all errors * remote debug info * Handle the different invocation cases * Handle non string properties * specify types to be analyzed * a bunch of tests * clean up * rasp lfi subs delayed (#4715) * Delay Appsec fs plugin subscription to fs:operations until the first req is received * disable rasp in tests * fix tests recursive call * Avoid multiple subscriptions to incomingHttpRequestStart * another try * replace spy with stub * execute unsubscribe asynchronously * sinon.assert async * clarify comment * Use a constant * Do not enable rasp in some tests * Remove not needed config property * Rename properties * Test iast and rasp fs-plugin subscription order * Avoid multiple analyzeLfi subscriptions * Block synchronous operations * Include synchronous blocking integration test * Test refactor * rename test file * Cleanup
* Report WAF fingerprints * WAF fingerprint RC capabilities * Linting * Remove useless file * Add blank line * Remove unused capability * Generate fingerprint on user login events * Fix linting * Add passport plugin test to GHA * Add business logic addressses * Add body-parser dep to passport plugin test * Reformat test * Refactor report derivatives * Move method to its right place * Unify reportSchemas and reportFingerprint test in one suite * Unify reportSchemas and reportFingerprint test in one suite
Allows for cheapest sampling context updates on async context switches and opens the path for profiling custom context.
…s when using `winston` (#4762)
* adds azure functions plugin * adds azure_functions plugin to API documentation * add typescript test for azure functions plugin * adds integration test for azure-functions plugin * add licenses for added dev packages * add azure-functions plugin to github workflow * use pipe for azure-functions integration test child process * update azure-functions integration test api route * refactor azure-functions integration test * add azure func command to path * remove yarn.lock file from azure-functions integration test * allow span kind to be server for azure functions * Update index.d.ts Co-authored-by: Roch Devost <roch.devost@datadoghq.com> * add serverless util * use built in url parser * remove serverless logic from web util * remove wait-on dependency * remove find-process dependency * Revert "remove find-process dependency" This reverts commit 3c004c5. * call func start directly and remove find-process dependency * simplify serverless util * Revert "simplify serverless util" This reverts commit 91a2dd9. * simplify serverless util --------- Co-authored-by: Roch Devost <roch.devost@datadoghq.com>
This commit does two things: - It lays the groundwork for an upcoming feature called "Code Origin for Spans". - To showcase this feature, it adds limited support for just Fastify entry-spans. To enable, set `DD_CODE_ORIGIN_FOR_SPANS_ENABLED=true`.
Split inspected code into multiple files. This makes adding new tests easier, as changes related to one inspected file doesn't influence unrelated tests.
* Path Parameters blocking * Lint * Change expect to assert in SRB tests * Change expect to assert in API Sec tests * Improve test naming * Correct spacing in tests Co-authored-by: Ugaitz Urien <ugaitz.urien@datadoghq.com> * Keep consistency of order in appsec channels * Better wrap fn naming in express instrumentation * Keep consistency of order in appsec channels handlers * Keep consistency of order in appsec channels handlers - test * Refactor express plugin test - use axios.create and getPort * Update packages/datadog-instrumentations/src/express.js Co-authored-by: simon-id <simon.id@datadoghq.com> --------- Co-authored-by: Ugaitz Urien <ugaitz.urien@datadoghq.com> Co-authored-by: simon-id <simon.id@datadoghq.com>
* vendor jsonpath-plus We need the latest version (10.0.0) so that it doesn't have vulnerabilities, but we need it to be compatible with Node.js 16.0.0, so we needed to vendor it and make slight adjustments. * more clarity in comment, and add the license
Overall package sizeSelf size: 7.55 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | @datadog/native-appsec | 8.1.1 | 18.67 MB | 18.68 MB | | @datadog/native-iast-taint-tracking | 3.1.0 | 12.27 MB | 12.28 MB | | @datadog/pprof | 5.3.0 | 9.85 MB | 10.22 MB | | protobufjs | 7.2.5 | 2.77 MB | 7.01 MB | | @datadog/native-iast-rewriter | 2.5.0 | 2.51 MB | 2.59 MB | | @opentelemetry/core | 1.14.0 | 872.87 kB | 1.47 MB | | @datadog/native-metrics | 2.0.0 | 898.77 kB | 1.3 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | import-in-the-middle | 1.11.2 | 112.74 kB | 826.22 kB | | msgpack-lite | 0.1.26 | 201.16 kB | 281.59 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.0 | 109.9 kB | 109.9 kB | | semver | 7.6.3 | 95.82 kB | 95.82 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | lru-cache | 7.14.0 | 74.95 kB | 74.95 kB | | ignore | 5.3.1 | 51.46 kB | 51.46 kB | | int64-buffer | 0.1.10 | 49.18 kB | 49.18 kB | | shell-quote | 1.8.1 | 44.96 kB | 44.96 kB | | istanbul-lib-coverage | 3.2.0 | 29.34 kB | 29.34 kB | | rfdc | 1.3.1 | 25.21 kB | 25.21 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | dc-polyfill | 0.1.4 | 23.1 kB | 23.1 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | path-to-regexp | 0.1.10 | 6.38 kB | 6.38 kB | | module-details-from-path | 1.0.3 | 4.47 kB | 4.47 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
simon-id
previously approved these changes
Oct 16, 2024
juan-fernandez
previously approved these changes
Oct 16, 2024
There was a problem hiding this comment.
looks good from test visibility's perspective. Could we please remove this line from the release notes though:
[fd0f5705fb] - (SEMVER-PATCH) [test visibility] Fix num tests reported by EFD (Juan Antonio Fernández de Alba) #4783
* update sinon * remove core setup from init integration test it's not needed, and our version of sinon doesn't work on node 12
rochdev
force-pushed
the
v4.48.0-proposal
branch
from
7458a6b
to
b9b7b76
Compare
BenchmarksBenchmark execution time: 2024-10-16 21:00:51 Comparing candidate commit b9b7b76 in PR branch Found 21 performance improvements and 24 performance regressions! Performance is the same for 730 metrics, 23 unstable metrics. scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-16
scenario:appsec-iast-no-vulnerability-iast-enabled-always-active-20
scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-16
scenario:appsec-iast-no-vulnerability-iast-enabled-default-config-20
scenario:appsec-iast-startup-time-iast-enabled-16
scenario:appsec-iast-startup-time-iast-enabled-18
scenario:appsec-iast-startup-time-iast-enabled-20
scenario:appsec-iast-with-vulnerability-iast-enabled-always-active-16
scenario:appsec-iast-with-vulnerability-iast-enabled-default-config-16
scenario:appsec-iast-with-vulnerability-iast-enabled-default-config-20
scenario:plugin-graphql-with-depth-and-collapse-off-16
scenario:plugin-graphql-with-depth-and-collapse-off-18
scenario:plugin-graphql-with-depth-and-collapse-off-20
scenario:plugin-graphql-with-depth-and-collapse-on-16
scenario:plugin-graphql-with-depth-and-collapse-on-18
scenario:plugin-graphql-with-depth-and-collapse-on-20
scenario:plugin-graphql-with-depth-off-16
scenario:plugin-graphql-with-depth-off-18
scenario:plugin-graphql-with-depth-off-20
scenario:plugin-graphql-with-depth-on-max-16
scenario:plugin-graphql-with-depth-on-max-18
scenario:plugin-graphql-with-depth-on-max-20
scenario:scope-manager-async_hooks-20
scenario:scope-manager-async_local_storage-20
scenario:scope-manager-async_resource-20
scenario:scope-manager-base-20
|
juan-fernandez
approved these changes
Oct 17, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
89619bdf46] - (SEMVER-PATCH) update body-parser (Bryan English) #4790d7b1dad805] - (SEMVER-PATCH) pin latest to 22 (Bryan English) #4793f8515ec28b] - (SEMVER-PATCH) Remove old debug option from docs (simon-id) #478659eb9a724a] - (SEMVER-PATCH) Don't stop the profiler if encoding a profile fails (Attila Szegedi) #47798969e05336] - (SEMVER-PATCH) vendor jsonpath-plus (Bryan English) #4785501ff2fbfb] - (SEMVER-MINOR) Suspicious request blocking - Express Path Parameters (Carles Capell) #4769944f57d5d4] - (SEMVER-PATCH) [DI] Refactor unit tests (Thomas Watson) #4777f62cbfadc7] - (SEMVER-PATCH) Unsubscribe NextJS body and query channels on appsec disable (Carles Capell) #4776c085df1eae] - (SEMVER-MINOR) Add support for Fastify entry spans for Code Origin for Spans (Thomas Watson) #4449bd4aff563f] - (SEMVER-MINOR) Update waf rules to 1.13.1 (Ugaitz Urien) #47685a113b2bcd] - (SEMVER-MINOR) Add Plugin for @azure/functions (Duncan Harvey) #4716ce0bdcea6e] - (SEMVER-MINOR) Fix capability identifier (Igor Unanua) #476760529442d2] - (SEMVER-MINOR) Use static vulnerability hash source when the cookie name is too long (Ugaitz Urien) #47645eea208392] - (SEMVER-MINOR) [test visibility] Add option to automatically report logs within tests when usingwinston(Juan Antonio Fernández de Alba) #47622d175d30d5] - (SEMVER-MINOR) Keep a profiling context object in spans (Attila Szegedi) #4763a2b318df27] - (SEMVER-MINOR) [ASM] Add support for attacker fingerprinting (Carles Capell) #4698111a156693] - (SEMVER-PATCH) Exploit Prevention LFI (Igor Unanua) #4715a11a1fd20e] - (SEMVER-MINOR) Upgrade iast rewriter to 2.5.0 (Igor Unanua) #47617f93d36b79] - (SEMVER-PATCH) use AsyncLocalStorage instead of our home-grown solutions (Bryan English) #4201bba5f3ddb3] - (SEMVER-MINOR) feat(dsm): implement avro schemas for avsc package (William Conti) #472608525d4c3c] - (SEMVER-MINOR) feat(tracing): implement protobufjs DSM schema support (William Conti) #4701d024777515] - (SEMVER-MINOR) [DI] Add ability to take state snapshot feature (Thomas Watson) #4549a00c9c8361] - (SEMVER-MINOR) Sql injection Exploit Prevention implementation for mysql2 library (Ugaitz Urien) #4712d1abcab7a1] - (SEMVER-MINOR) [DI] Add hostname to probe result (Thomas Watson) #4756d1f29dba99] - (SEMVER-PATCH) Fix appsec rate limiter flaky test (Ugaitz Urien) #4754eef6711411] - (SEMVER-PATCH) Fix child process not maintaining previous parent span after execution (Ugaitz Urien) #4752c700341689] - (SEMVER-PATCH) prefix system-tests env var names (William Conti) #4746f988e003bf] - (SEMVER-MINOR) [DI] Add GitHub repo and SHA tags to probe results (Thomas Watson) #4751e09305d366] - (SEMVER-PATCH) [DI] Fix probe.location.lines to be string[] instead of number[] (Thomas Watson) #4750748ef616c3] - (SEMVER-PATCH) [DI] Switch unit tests to Mocha instead of Tap (Thomas Watson) #472870d5591d9b] - (SEMVER-MINOR) [test visibility] Readpull_requestandpull_request_targetevent info from GHA (Juan Antonio Fernández de Alba) #47454d2f5b86a0] - (SEMVER-PATCH) Don't use deprecated url.parse function (Thomas Watson) #474392515a65e3] - (SEMVER-MINOR) [DI] Add stack trace to log probe results (Thomas Watson) #4727