[ASM]Fix appsec waf benchmark for real #6329

anna-git · 2024-11-20T17:24:28Z

Summary of changes

Don't throw when creating a trace context and an empty datadog tracer

Reason for change

It was throwing when creating the context.

Implementation details

Test coverage

Other details

andrewlock · 2024-11-20T17:56:22Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6329) - mean (80ms)  : 66, 93
     .   : milestone, 80,
    master - mean (72ms)  : 63, 82
     .   : milestone, 72,

    section CallTarget+Inlining+NGEN
    This PR (6329) - mean (1,182ms)  : crit, 1023, 1341
     .   : crit, milestone, 1182,
    master - mean (1,108ms)  : 1092, 1123
     .   : milestone, 1108,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6329) - mean (113ms)  : 109, 116
     .   : milestone, 113,
    master - mean (108ms)  : 106, 110
     .   : milestone, 108,

    section CallTarget+Inlining+NGEN
    This PR (6329) - mean (813ms)  : 729, 896
     .   : milestone, 813,
    master - mean (767ms)  : 751, 783
     .   : milestone, 767,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6329) - mean (98ms)  : 85, 111
     .   : milestone, 98,
    master - mean (92ms)  : 90, 94
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (6329) - mean (755ms)  : 677, 832
     .   : milestone, 755,
    master - mean (729ms)  : 714, 743
     .   : milestone, 729,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6329) - mean (191ms)  : 186, 197
     .   : milestone, 191,
    master - mean (191ms)  : 186, 196
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (6329) - mean (1,213ms)  : 1192, 1233
     .   : milestone, 1213,
    master - mean (1,209ms)  : 1187, 1231
     .   : milestone, 1209,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6329) - mean (277ms)  : 273, 281
     .   : milestone, 277,
    master - mean (276ms)  : 272, 279
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (6329) - mean (941ms)  : 922, 959
     .   : milestone, 941,
    master - mean (945ms)  : 923, 966
     .   : milestone, 945,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6329) - mean (266ms)  : 260, 272
     .   : milestone, 266,
    master - mean (267ms)  : 264, 271
     .   : milestone, 267,

    section CallTarget+Inlining+NGEN
    This PR (6329) - mean (927ms)  : 908, 946
     .   : milestone, 927,
    master - mean (931ms)  : 912, 951
     .   : milestone, 931,

link04

Works for me too!
Would also recommend copyright removing the extra copyright, just noticed it.

datadog-ddstaging · 2024-11-20T18:25:24Z

Datadog Report

Branch report: anna/asm/fix-waf-benchmarks-tracecontext-null
Commit report: b1e9ab9
Test service: dd-trace-dotnet

✅ 0 Failed, 451605 Passed, 2767 Skipped, 20h 29m 13.79s Total Time
❄️ 2 New Flaky

New Flaky Tests (2)

OnSupportedFrameworkInSsi_CallsForwarderWithExpectedTelemetry - Datadog.Trace.ClrProfiler.IntegrationTests.InstrumentationTests - Last Failure
Expand for error
```
 Expected File.Exists(echoLogFileName) to be true, but found False.
```
OnSupportedFrameworkInSsi_CallsForwarderWithExpectedTelemetry - Datadog.Trace.ClrProfiler.IntegrationTests.InstrumentationTests - Last Failure
Expand for error
```
 Expected File.Exists(_appPath) to be true, but found False.
```

andrewlock · 2024-11-21T03:07:55Z

Benchmarks Report for tracer 🐌

Benchmarks for #6329 compared to master:

4 benchmarks are faster, with geometric mean 1.176
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.01μs	40.6ns	177ns	0.012	0.00399	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.1μs	52.5ns	241ns	0.0262	0.0157	0.00523	5.8 KB
master	`StartStopWithChild`	net472	16.4μs	51.6ns	193ns	1.03	0.301	0.0977	6.21 KB
#6329	`StartStopWithChild`	net6.0	8.16μs	45.3ns	286ns	0.0123	0.0041	0	5.61 KB
#6329	`StartStopWithChild`	netcoreapp3.1	9.98μs	53.7ns	289ns	0.0143	0.00476	0	5.8 KB
#6329	`StartStopWithChild`	net472	16.2μs	75.5ns	302ns	1.04	0.299	0.097	6.21 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	501μs	250ns	968ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	645μs	335ns	1.3μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	873μs	454ns	1.7μs	0.434	3.3 KB
#6329	`WriteAndFlushEnrichedTraces`	net6.0	482μs	320ns	1.24μs	0	2.7 KB
#6329	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	653μs	486ns	1.88μs	0	2.7 KB
#6329	`WriteAndFlushEnrichedTraces`	net472	857μs	343ns	1.33μs	0.428	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	200μs	1.16μs	9.1μs	0.2	18.73 KB
master	`SendRequest`	netcoreapp3.1	222μs	1.25μs	9.29μs	0.222	20.89 KB
master	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
#6329	`SendRequest`	net6.0	199μs	1.15μs	10.7μs	0.2	18.73 KB
#6329	`SendRequest`	netcoreapp3.1	222μs	1.26μs	10.2μs	0.215	20.89 KB
#6329	`SendRequest`	net472	0.00242ns	0.00107ns	0.004ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	577μs	2.92μs	16.8μs	0.553	0	0	41.76 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	675μs	3μs	11.6μs	0.336	0	0	41.83 KB
master	`WriteAndFlushEnrichedTraces`	net472	868μs	3.23μs	12.5μs	8.3	2.62	0.437	53.28 KB
#6329	`WriteAndFlushEnrichedTraces`	net6.0	568μs	2.44μs	9.43μs	0.579	0	0	41.8 KB
#6329	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	679μs	3.51μs	16.1μs	0.336	0	0	41.84 KB
#6329	`WriteAndFlushEnrichedTraces`	net472	869μs	3.38μs	12.6μs	8.08	2.55	0.425	53.29 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.33μs	1.03ns	3.84ns	0.014	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.77μs	3.27ns	12.7ns	0.0131	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.09μs	1.04ns	3.75ns	0.156	0.00104	987 B
#6329	`ExecuteNonQuery`	net6.0	1.31μs	1.15ns	4.45ns	0.0143	0	1.02 KB
#6329	`ExecuteNonQuery`	netcoreapp3.1	1.73μs	0.916ns	3.43ns	0.0137	0	1.02 KB
#6329	`ExecuteNonQuery`	net472	2.04μs	1.57ns	5.88ns	0.156	0.00101	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.28μs	0.482ns	1.87ns	0.0135	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.54μs	1.03ns	3.99ns	0.0129	976 B
master	`CallElasticsearch`	net472	2.51μs	1.16ns	4.5ns	0.157	995 B
master	`CallElasticsearchAsync`	net6.0	1.28μs	0.633ns	2.45ns	0.0135	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.65μs	0.898ns	3.36ns	0.014	1.02 KB
master	`CallElasticsearchAsync`	net472	2.56μs	0.795ns	2.98ns	0.167	1.05 KB
#6329	`CallElasticsearch`	net6.0	1.19μs	0.516ns	1.93ns	0.0138	976 B
#6329	`CallElasticsearch`	netcoreapp3.1	1.52μs	1.92ns	7.18ns	0.0131	976 B
#6329	`CallElasticsearch`	net472	2.59μs	0.691ns	2.49ns	0.158	995 B
#6329	`CallElasticsearchAsync`	net6.0	1.31μs	0.655ns	2.45ns	0.0131	952 B
#6329	`CallElasticsearchAsync`	netcoreapp3.1	1.71μs	0.835ns	3.23ns	0.0135	1.02 KB
#6329	`CallElasticsearchAsync`	net472	2.6μs	0.978ns	3.66ns	0.167	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6329

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net6.0	1.270	1,508.61	1,187.56

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.51μs	0.534ns	2.07ns	0.0134	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.59μs	0.857ns	3.2ns	0.0127	952 B
master	`ExecuteAsync`	net472	1.8μs	0.547ns	2.12ns	0.145	915 B
#6329	`ExecuteAsync`	net6.0	1.19μs	0.559ns	2.09ns	0.0133	952 B
#6329	`ExecuteAsync`	netcoreapp3.1	1.63μs	0.682ns	2.64ns	0.0129	952 B
#6329	`ExecuteAsync`	net472	1.85μs	1.61ns	6.22ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.3μs	3.14ns	12.2ns	0.0325	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.23μs	5.65ns	21.9ns	0.0368	2.85 KB
master	`SendAsync`	net472	7.23μs	8.75ns	33.9ns	0.494	3.12 KB
#6329	`SendAsync`	net6.0	4.35μs	1.65ns	6.39ns	0.0326	2.31 KB
#6329	`SendAsync`	netcoreapp3.1	5.36μs	5.19ns	20.1ns	0.0378	2.85 KB
#6329	`SendAsync`	net472	7.33μs	2.31ns	8.93ns	0.495	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6329

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog‑net472	1.129	2,696.02	2,387.62

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.58μs	0.613ns	2.3ns	0.0232	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.19μs	1.32ns	4.76ns	0.0221	1.64 KB
master	`EnrichedLog`	net472	2.7μs	5.51ns	21.3ns	0.25	1.57 KB
#6329	`EnrichedLog`	net6.0	1.46μs	0.96ns	3.59ns	0.0227	1.64 KB
#6329	`EnrichedLog`	netcoreapp3.1	2.17μs	1.95ns	7.56ns	0.0222	1.64 KB
#6329	`EnrichedLog`	net472	2.39μs	1.04ns	3.88ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	119μs	237ns	918ns	0.06	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	124μs	114ns	443ns	0	0	4.28 KB
master	`EnrichedLog`	net472	152μs	187ns	725ns	0.687	0.229	4.46 KB
#6329	`EnrichedLog`	net6.0	120μs	97.2ns	364ns	0	0	4.28 KB
#6329	`EnrichedLog`	netcoreapp3.1	123μs	195ns	757ns	0.0616	0	4.28 KB
#6329	`EnrichedLog`	net472	152μs	95ns	368ns	0.679	0.226	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.07μs	2.25ns	8.73ns	0.0306	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.01μs	1.44ns	5.57ns	0.03	2.2 KB
master	`EnrichedLog`	net472	4.88μs	5.97ns	23.1ns	0.32	2.02 KB
#6329	`EnrichedLog`	net6.0	2.98μs	1.68ns	6.52ns	0.0312	2.2 KB
#6329	`EnrichedLog`	netcoreapp3.1	4.31μs	1.18ns	4.58ns	0.0302	2.2 KB
#6329	`EnrichedLog`	net472	4.89μs	2.51ns	9.71ns	0.32	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.32μs	0.646ns	2.5ns	0.0158	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.73μs	1.12ns	4.34ns	0.0148	1.14 KB
master	`SendReceive`	net472	1.99μs	1.25ns	4.84ns	0.183	1.16 KB
#6329	`SendReceive`	net6.0	1.41μs	0.558ns	2.16ns	0.0164	1.14 KB
#6329	`SendReceive`	netcoreapp3.1	1.76μs	1.82ns	7.06ns	0.0149	1.14 KB
#6329	`SendReceive`	net472	2.09μs	1.57ns	6.07ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.95μs	0.924ns	3.46ns	0.022	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.85μs	2.7ns	10.5ns	0.021	1.65 KB
master	`EnrichedLog`	net472	4.3μs	1.44ns	5.4ns	0.323	2.04 KB
#6329	`EnrichedLog`	net6.0	2.79μs	1.7ns	6.59ns	0.0224	1.6 KB
#6329	`EnrichedLog`	netcoreapp3.1	4.01μs	2.08ns	8.07ns	0.0221	1.65 KB
#6329	`EnrichedLog`	net472	4.39μs	10.7ns	41.4ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6329

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0	1.182	493.56	417.56

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	493ns	0.527ns	2.04ns	0.00801	576 B
master	`StartFinishSpan`	netcoreapp3.1	552ns	0.94ns	3.64ns	0.00779	576 B
master	`StartFinishSpan`	net472	636ns	1.38ns	5.33ns	0.0917	578 B
master	`StartFinishScope`	net6.0	488ns	0.365ns	1.42ns	0.00966	696 B
master	`StartFinishScope`	netcoreapp3.1	671ns	0.68ns	2.63ns	0.00929	696 B
master	`StartFinishScope`	net472	971ns	3.45ns	13.4ns	0.104	658 B
#6329	`StartFinishSpan`	net6.0	418ns	0.359ns	1.39ns	0.00814	576 B
#6329	`StartFinishSpan`	netcoreapp3.1	594ns	0.619ns	2.4ns	0.00791	576 B
#6329	`StartFinishSpan`	net472	661ns	0.828ns	3.1ns	0.0918	578 B
#6329	`StartFinishScope`	net6.0	495ns	0.61ns	2.36ns	0.00978	696 B
#6329	`StartFinishScope`	netcoreapp3.1	710ns	1.08ns	4.17ns	0.00947	696 B
#6329	`StartFinishScope`	net472	873ns	1.52ns	5.87ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6329

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.130	677.49	599.66

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	678ns	1.39ns	5.37ns	0.00992	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	966ns	0.736ns	2.85ns	0.00918	696 B
master	`RunOnMethodBegin`	net472	1.1μs	2.28ns	8.83ns	0.104	658 B
#6329	`RunOnMethodBegin`	net6.0	600ns	1.08ns	4.17ns	0.00991	696 B
#6329	`RunOnMethodBegin`	netcoreapp3.1	907ns	2.37ns	9.2ns	0.00918	696 B
#6329	`RunOnMethodBegin`	net472	1.16μs	1.58ns	5.92ns	0.104	658 B

Dont throw please

b1e9ab9

anna-git requested a review from a team as a code owner November 20, 2024 17:24

github-actions bot added area:tests unit tests, integration tests area:asm labels Nov 20, 2024

link04 approved these changes Nov 20, 2024

View reviewed changes

NachoEchevarria approved these changes Nov 21, 2024

View reviewed changes

anna-git merged commit b6e8358 into master Nov 21, 2024
76 of 78 checks passed

anna-git deleted the anna/asm/fix-waf-benchmarks-tracecontext-null branch November 21, 2024 10:17

github-actions bot added this to the vNext-v3 milestone Nov 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ASM]Fix appsec waf benchmark for real #6329

[ASM]Fix appsec waf benchmark for real #6329

anna-git commented Nov 20, 2024 •

edited

Loading

andrewlock commented Nov 20, 2024

link04 left a comment

datadog-ddstaging bot commented Nov 20, 2024 •

edited

Loading

andrewlock commented Nov 21, 2024

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6329

Raw results

Raw results

Faster 🎉 in #6329

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6329

Raw results

Faster 🎉 in #6329

Raw results

[ASM]Fix appsec waf benchmark for real #6329

[ASM]Fix appsec waf benchmark for real #6329

Conversation

anna-git commented Nov 20, 2024 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

andrewlock commented Nov 20, 2024

Execution-Time Benchmarks Report ⏱️

link04 left a comment

Choose a reason for hiding this comment

datadog-ddstaging bot commented Nov 20, 2024 • edited Loading

Datadog Report

New Flaky Tests (2)

andrewlock commented Nov 21, 2024

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6329

Raw results

Raw results

Faster 🎉 in #6329

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6329

Raw results

Faster 🎉 in #6329

Raw results

anna-git commented Nov 20, 2024 •

edited

Loading

datadog-ddstaging bot commented Nov 20, 2024 •

edited

Loading