MVP for test containers (with aerospike) #5031

kevingosse · 2024-01-08T15:00:53Z

Summary of changes

Add testcontainers to our integration tests. Convert the Aerospike tests to use testcontainers.

Reason for change

Much easier to launch the tests locally.

Implementation details

For each service to run in docker, an instance of ContainerFixture must be implemented to explain what image to use and how to configure it. Then tests that depend on it must import the fixture.

In the xunit executor, we find all the docker fixtures needed to run the set of tests, and we start them. They're cleaned up after all tests cases have run (for future improvement, we can try to stop the docker images as soon as they're not needed anymore).
Because fixtures are not singleton, the ContainersRegistry is used to store the objects across multiple fixtures.

Test coverage

Only Aerospike is converted in this PR.

datadog-ddstaging · 2024-01-08T15:25:09Z

Datadog Report

Branch report: kevin/test-containers-mvp
Commit report: 55f2ea0
Test service: dd-trace-dotnet

✅ 0 Failed, 306545 Passed, 1528 Skipped, 1h 0m 31.38s Wall Time

andrewlock · 2024-01-08T15:36:33Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5031) - mean (69ms)  : 66, 71
     .   : milestone, 69,
    master - mean (68ms)  : 66, 71
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (5031) - mean (976ms)  : 956, 995
     .   : milestone, 976,
    master - mean (979ms)  : 948, 1010
     .   : milestone, 979,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5031) - mean (107ms)  : 105, 109
     .   : milestone, 107,
    master - mean (107ms)  : 105, 109
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (5031) - mean (673ms)  : 655, 691
     .   : milestone, 673,
    master - mean (678ms)  : 664, 693
     .   : milestone, 678,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5031) - mean (91ms)  : 89, 92
     .   : milestone, 91,
    master - mean (91ms)  : 89, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (5031) - mean (627ms)  : 612, 642
     .   : milestone, 627,
    master - mean (635ms)  : 620, 649
     .   : milestone, 635,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5031) - mean (193ms)  : 189, 198
     .   : milestone, 193,
    master - mean (194ms)  : 190, 198
     .   : milestone, 194,

    section CallTarget+Inlining+NGEN
    This PR (5031) - mean (1,103ms)  : 1077, 1129
     .   : milestone, 1103,
    master - mean (1,103ms)  : 1066, 1140
     .   : milestone, 1103,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5031) - mean (278ms)  : 273, 283
     .   : milestone, 278,
    master - mean (277ms)  : 274, 281
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (5031) - mean (872ms)  : 842, 902
     .   : milestone, 872,
    master - mean (870ms)  : 843, 897
     .   : milestone, 870,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5031) - mean (266ms)  : 262, 270
     .   : milestone, 266,
    master - mean (267ms)  : 263, 271
     .   : milestone, 267,

    section CallTarget+Inlining+NGEN
    This PR (5031) - mean (843ms)  : 809, 877
     .   : milestone, 843,
    master - mean (852ms)  : 825, 878
     .   : milestone, 852,

andrewlock · 2024-01-08T19:35:30Z

Benchmarks Report 🐌

Benchmarks for #5031 compared to master:

1 benchmarks are faster, with geometric mean 1.136
2 benchmarks are slower, with geometric mean 1.166
1 benchmarks have fewer allocations
2 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.6μs	45ns	221ns	0.0219	0.00876	0	7.48 KB
master	`StartStopWithChild`	netcoreapp3.1	10.9μs	56ns	306ns	0.0266	0.0106	0	7.58 KB
master	`StartStopWithChild`	net472	17.3μs	48.5ns	188ns	1.32	0.331	0.0957	7.96 KB
#5031	`StartStopWithChild`	net6.0	8.81μs	46.2ns	226ns	0.0355	0.0178	0	7.47 KB
#5031	`StartStopWithChild`	netcoreapp3.1	10.7μs	57.8ns	306ns	0.031	0.0155	0	7.58 KB
#5031	`StartStopWithChild`	net472	17.3μs	48.3ns	187ns	1.32	0.335	0.112	7.95 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	445μs	220ns	850ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	616μs	293ns	1.14μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	794μs	337ns	1.31μs	0.396	3.3 KB
#5031	`WriteAndFlushEnrichedTraces`	net6.0	454μs	446ns	1.73μs	0	2.7 KB
#5031	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	614μs	208ns	804ns	0	2.7 KB
#5031	`WriteAndFlushEnrichedTraces`	net472	781μs	264ns	989ns	0.391	3.3 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5031

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1	1.136	226.45	199.32

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	38.8μs	22.5ns	87ns	0.0197	0	2.39 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	41.3μs	42.5ns	159ns	0.0207	0	2.37 KB
master	`AllCycleSimpleBody`	net472	45.1μs	20.5ns	76.6ns	0.385	0	2.45 KB
master	`AllCycleMoreComplexBody`	net6.0	199μs	79.6ns	308ns	0.0992	0	9.87 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	211μs	285ns	1.1μs	0.106	0	9.76 KB
master	`AllCycleMoreComplexBody`	net472	248μs	96.5ns	361ns	1.49	0	9.95 KB
master	`ObjectExtractorSimpleBody`	net6.0	144ns	0.112ns	0.436ns	0.00393	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	227ns	0.159ns	0.595ns	0.00368	0	272 B
master	`ObjectExtractorSimpleBody`	net472	165ns	0.124ns	0.462ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	3.04μs	1.68ns	6.04ns	0.0532	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	3.94μs	3.37ns	13.1ns	0.0512	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	3.75μs	2.44ns	9.43ns	0.602	0.00559	3.8 KB
#5031	`AllCycleSimpleBody`	net6.0	39.6μs	47.9ns	185ns	0.0197	0	2.39 KB
#5031	`AllCycleSimpleBody`	netcoreapp3.1	42.3μs	64.9ns	251ns	0.0211	0	2.37 KB
#5031	`AllCycleSimpleBody`	net472	45.8μs	10ns	37.4ns	0.387	0	2.45 KB
#5031	`AllCycleMoreComplexBody`	net6.0	202μs	204ns	790ns	0.101	0	9.87 KB
#5031	`AllCycleMoreComplexBody`	netcoreapp3.1	213μs	324ns	1.26μs	0.105	0	9.76 KB
#5031	`AllCycleMoreComplexBody`	net472	230μs	120ns	464ns	1.49	0	9.95 KB
#5031	`ObjectExtractorSimpleBody`	net6.0	142ns	0.16ns	0.598ns	0.00394	0	280 B
#5031	`ObjectExtractorSimpleBody`	netcoreapp3.1	200ns	0.201ns	0.78ns	0.0037	0	272 B
#5031	`ObjectExtractorSimpleBody`	net472	182ns	0.253ns	0.98ns	0.0446	0	281 B
#5031	`ObjectExtractorMoreComplexBody`	net6.0	3.01μs	2.48ns	9.29ns	0.0525	0	3.78 KB
#5031	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.02μs	1.45ns	5.21ns	0.0503	0	3.69 KB
#5031	`ObjectExtractorMoreComplexBody`	net472	3.75μs	2.14ns	7.73ns	0.603	0.0056	3.8 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`RunWaf(args=NestedMap (10))`	net6.0	59μs	71.5ns	267ns	0.234	0	16.69 KB
master	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	70.9μs	201ns	779ns	0.216	0	16.68 KB
master	`RunWaf(args=NestedMap (10))`	net472	103μs	216ns	837ns	2.65	0.0983	16.77 KB
master	`RunWafTwice(args=NestedMap (10))`	net6.0	59.7μs	346ns	2.94μs	0.246	0	17.39 KB
master	`RunWafTwice(args=NestedMap (10))`	netcoreapp3.1	72.5μs	386ns	1.93μs	0.228	0	17.38 KB
master	`RunWafTwice(args=NestedMap (10))`	net472	109μs	498ns	1.93μs	2.76	0.11	17.5 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	116μs	530ns	2.05μs	0.328	0	23.03 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	129μs	135ns	486ns	0.257	0	22.98 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	166μs	414ns	1.49μs	3.67	0.16	23.33 KB
master	`RunWaf(args=NestedMap (100))`	net6.0	101μs	24.9ns	93.1ns	0.452	0	33.38 KB
master	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	141μs	779ns	4.54μs	0.425	0	33.95 KB
master	`RunWaf(args=NestedMap (100))`	net472	192μs	327ns	1.18μs	5.38	0.472	34.31 KB
master	`RunWafTwice(args=NestedMap (100))`	net6.0	104μs	584ns	3.6μs	0.436	0	34.09 KB
master	`RunWafTwice(args=NestedMap (100))`	netcoreapp3.1	143μs	616ns	2.31μs	0.408	0	34.65 KB
master	`RunWafTwice(args=NestedMap (100))`	net472	198μs	163ns	632ns	5.5	0.393	35.03 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	164μs	58.5ns	211ns	0.491	0	39.73 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	200μs	106ns	367ns	0.503	0	40.26 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	261μs	369ns	1.43μs	6.44	0.515	40.87 KB
master	`RunWaf(args=NestedMap (20))`	net6.0	105μs	27.3ns	98.5ns	0.45	0	32.81 KB
master	`RunWaf(args=NestedMap (20))`	netcoreapp3.1	137μs	714ns	3.42μs	0.397	0	32.92 KB
master	`RunWaf(args=NestedMap (20))`	net472	188μs	93.4ns	362ns	5.24	0.374	33.27 KB
master	`RunWafTwice(args=NestedMap (20))`	net6.0	105μs	160ns	619ns	0.47	0	33.51 KB
master	`RunWafTwice(args=NestedMap (20))`	netcoreapp3.1	141μs	240ns	930ns	0.423	0	33.62 KB
master	`RunWafTwice(args=NestedMap (20))`	net472	195μs	115ns	399ns	5.39	0.392	34 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	164μs	61.7ns	239ns	0.516	0	39.15 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	202μs	564ns	2.18μs	0.509	0	39.22 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	260μs	300ns	1.16μs	6.22	0.508	39.83 KB
#5031	`RunWaf(args=NestedMap (10))`	net6.0	59.1μs	22.2ns	85.9ns	0.235	0	16.69 KB
#5031	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	70.2μs	385ns	2.18μs	0.211	0	16.68 KB
#5031	`RunWaf(args=NestedMap (10))`	net472	98.4μs	110ns	426ns	2.64	0.0977	16.77 KB
#5031	`RunWafTwice(args=NestedMap (10))`	net6.0	59.3μs	18.8ns	67.6ns	0.247	0	17.39 KB
#5031	`RunWafTwice(args=NestedMap (10))`	netcoreapp3.1	74.6μs	262ns	1.01μs	0.227	0	17.38 KB
#5031	`RunWafTwice(args=NestedMap (10))`	net472	107μs	53.8ns	201ns	2.73	0.105	17.5 KB
#5031	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	114μs	618ns	3.38μs	0.326	0	23.03 KB
#5031	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	132μs	539ns	2.02μs	0.256	0	22.99 KB
#5031	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	164μs	83.2ns	322ns	3.68	0.164	23.33 KB
#5031	`RunWaf(args=NestedMap (100))`	net6.0	109μs	617ns	4.23μs	0.446	0	33.38 KB
#5031	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	132μs	408ns	1.73μs	0.41	0	33.95 KB
#5031	`RunWaf(args=NestedMap (100))`	net472	197μs	724ns	2.8μs	5.42	0.451	34.31 KB
#5031	`RunWafTwice(args=NestedMap (100))`	net6.0	109μs	55.3ns	214ns	0.437	0	34.09 KB
#5031	`RunWafTwice(args=NestedMap (100))`	netcoreapp3.1	145μs	690ns	2.85μs	0.442	0	34.65 KB
#5031	`RunWafTwice(args=NestedMap (100))`	net472	196μs	144ns	556ns	5.49	0.392	35.04 KB
#5031	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	166μs	59.2ns	221ns	0.567	0	39.73 KB
#5031	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	202μs	1.08μs	5.63μs	0.507	0	40.26 KB
#5031	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	260μs	445ns	1.72μs	6.4	0.512	40.86 KB
#5031	`RunWaf(args=NestedMap (20))`	net6.0	101μs	525ns	2.46μs	0.446	0	32.81 KB
#5031	`RunWaf(args=NestedMap (20))`	netcoreapp3.1	138μs	666ns	2.75μs	0.413	0	32.92 KB
#5031	`RunWaf(args=NestedMap (20))`	net472	187μs	153ns	591ns	5.25	0.375	33.27 KB
#5031	`RunWafTwice(args=NestedMap (20))`	net6.0	113μs	363ns	1.41μs	0.455	0	33.51 KB
#5031	`RunWafTwice(args=NestedMap (20))`	netcoreapp3.1	139μs	647ns	2.5μs	0.422	0	33.62 KB
#5031	`RunWafTwice(args=NestedMap (20))`	net472	195μs	91.6ns	343ns	5.32	0.387	34 KB
#5031	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	171μs	389ns	1.51μs	0.493	0	39.15 KB
#5031	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	203μs	1.1μs	6.43μs	0.482	0	39.22 KB
#5031	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	257μs	221ns	858ns	6.26	0.511	39.83 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	173μs	130ns	485ns	0.173	18.25 KB
master	`SendRequest`	netcoreapp3.1	195μs	229ns	886ns	0.193	20.42 KB
master	`SendRequest`	net472	0.00136ns	0.000337ns	0.00131ns	0	0 b
#5031	`SendRequest`	net6.0	170μs	116ns	449ns	0.17	18.25 KB
#5031	`SendRequest`	netcoreapp3.1	193μs	222ns	859ns	0.192	20.41 KB
#5031	`SendRequest`	net472	0ns	0ns	0ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #5031

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0	41.44 KB	41.75 KB	316 B	0.76%
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1	41.68 KB	41.95 KB	267 B	0.64%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	549μs	1.03μs	4.01μs	0.556	0	0	41.44 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	663μs	1.49μs	5.78μs	0.327	0	0	41.68 KB
master	`WriteAndFlushEnrichedTraces`	net472	821μs	3.7μs	14.3μs	8.31	2.37	0.396	53.25 KB
#5031	`WriteAndFlushEnrichedTraces`	net6.0	539μs	1.94μs	7.52μs	0.532	0	0	41.75 KB
#5031	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	670μs	1.98μs	7.66μs	0.332	0	0	41.95 KB
#5031	`WriteAndFlushEnrichedTraces`	net472	847μs	3.69μs	14.3μs	8.22	2.47	0.411	53.21 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.13μs	0.611ns	2.37ns	0.0107	768 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.47μs	1.08ns	4.19ns	0.0104	768 B
master	`ExecuteNonQuery`	net472	1.79μs	1.23ns	4.76ns	0.116	730 B
#5031	`ExecuteNonQuery`	net6.0	1.06μs	0.639ns	2.47ns	0.0106	768 B
#5031	`ExecuteNonQuery`	netcoreapp3.1	1.43μs	0.511ns	1.91ns	0.01	768 B
#5031	`ExecuteNonQuery`	net472	1.81μs	1.24ns	4.81ns	0.115	730 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.15μs	0.386ns	1.45ns	0.0133	936 B
master	`CallElasticsearch`	netcoreapp3.1	1.51μs	0.916ns	3.43ns	0.013	936 B
master	`CallElasticsearch`	net472	2.54μs	0.722ns	2.7ns	0.151	955 B
master	`CallElasticsearchAsync`	net6.0	1.2μs	0.62ns	2.32ns	0.0126	912 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.58μs	1.12ns	4.34ns	0.0128	984 B
master	`CallElasticsearchAsync`	net472	2.71μs	1.19ns	4.61ns	0.16	1.01 KB
#5031	`CallElasticsearch`	net6.0	1.19μs	1.42ns	5.32ns	0.0127	936 B
#5031	`CallElasticsearch`	netcoreapp3.1	1.48μs	0.842ns	3.26ns	0.012	936 B
#5031	`CallElasticsearch`	net472	2.6μs	2.03ns	7.86ns	0.152	955 B
#5031	`CallElasticsearchAsync`	net6.0	1.33μs	0.604ns	2.26ns	0.0124	912 B
#5031	`CallElasticsearchAsync`	netcoreapp3.1	1.6μs	1.07ns	4.14ns	0.0135	984 B
#5031	`CallElasticsearchAsync`	net472	2.66μs	0.792ns	2.86ns	0.16	1.01 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.26μs	0.814ns	3.05ns	0.0126	912 B
master	`ExecuteAsync`	netcoreapp3.1	1.57μs	0.732ns	2.84ns	0.0118	912 B
master	`ExecuteAsync`	net472	1.92μs	1.07ns	4.15ns	0.138	875 B
#5031	`ExecuteAsync`	net6.0	1.3μs	0.666ns	2.49ns	0.013	912 B
#5031	`ExecuteAsync`	netcoreapp3.1	1.51μs	0.46ns	1.72ns	0.0121	912 B
#5031	`ExecuteAsync`	net472	1.82μs	1.67ns	6.48ns	0.139	875 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.13μs	1.29ns	5.01ns	0.0291	2.1 KB
master	`SendAsync`	netcoreapp3.1	4.95μs	2.04ns	7.64ns	0.0346	2.63 KB
master	`SendAsync`	net472	7.77μs	3.62ns	13ns	0.525	3.31 KB
#5031	`SendAsync`	net6.0	3.96μs	1.64ns	6.35ns	0.0298	2.1 KB
#5031	`SendAsync`	netcoreapp3.1	4.99μs	8.16ns	31.6ns	0.0348	2.63 KB
#5031	`SendAsync`	net472	7.83μs	2.51ns	9.05ns	0.525	3.31 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #5031

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472	62.12 KB	59.26 KB	-2.86 KB	-4.61%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark`	net6.0	62.5μs	1.04μs	10.2μs	43.44 KB
master	`StringConcatBenchmark`	netcoreapp3.1	62.4μs	800ns	7.92μs	42.64 KB
master	`StringConcatBenchmark`	net472	37.1μs	72.1ns	260ns	62.12 KB
master	`StringConcatAspectBenchmark`	net6.0	284μs	3.45μs	31μs	204.05 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	255μs	3.46μs	32.3μs	202.98 KB
master	`StringConcatAspectBenchmark`	net472	253μs	5.51μs	52.3μs	221.18 KB
#5031	`StringConcatBenchmark`	net6.0	61.2μs	981ns	9.35μs	43.44 KB
#5031	`StringConcatBenchmark`	netcoreapp3.1	54.3μs	184ns	665ns	42.64 KB
#5031	`StringConcatBenchmark`	net472	38.1μs	176ns	725ns	59.26 KB
#5031	`StringConcatAspectBenchmark`	net6.0	287μs	4.97μs	44.7μs	204.36 KB
#5031	`StringConcatAspectBenchmark`	netcoreapp3.1	284μs	2.89μs	26.5μs	202.73 KB
#5031	`StringConcatAspectBenchmark`	net472	255μs	6.07μs	58.2μs	221.18 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.43μs	0.566ns	2.12ns	0.0222	1.57 KB
master	`EnrichedLog`	netcoreapp3.1	2.15μs	1.21ns	4.67ns	0.0208	1.57 KB
master	`EnrichedLog`	net472	2.7μs	4.5ns	17.4ns	0.237	1.5 KB
#5031	`EnrichedLog`	net6.0	1.54μs	1.07ns	3.87ns	0.0223	1.57 KB
#5031	`EnrichedLog`	netcoreapp3.1	2.28μs	0.53ns	1.98ns	0.0217	1.57 KB
#5031	`EnrichedLog`	net472	2.62μs	1.81ns	6.76ns	0.238	1.5 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	113μs	134ns	518ns	0.0567	0	4.21 KB
master	`EnrichedLog`	netcoreapp3.1	117μs	124ns	481ns	0.0584	0	4.21 KB
master	`EnrichedLog`	net472	148μs	56.1ns	217ns	0.664	0.221	4.39 KB
#5031	`EnrichedLog`	net6.0	112μs	87.1ns	326ns	0.0563	0	4.21 KB
#5031	`EnrichedLog`	netcoreapp3.1	117μs	70.3ns	272ns	0	0	4.21 KB
#5031	`EnrichedLog`	net472	147μs	81.4ns	315ns	0.659	0.22	4.39 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3.13μs	0.84ns	3.25ns	0.0298	2.13 KB
master	`EnrichedLog`	netcoreapp3.1	4.18μs	1.4ns	5.41ns	0.0293	2.13 KB
master	`EnrichedLog`	net472	4.79μs	2.46ns	9.53ns	0.309	1.95 KB
#5031	`EnrichedLog`	net6.0	2.92μs	1.16ns	4.34ns	0.0293	2.13 KB
#5031	`EnrichedLog`	netcoreapp3.1	4.13μs	1.86ns	7.19ns	0.0286	2.13 KB
#5031	`EnrichedLog`	net472	4.92μs	2.46ns	9.19ns	0.309	1.95 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.26μs	0.708ns	2.74ns	0.0156	1.1 KB
master	`SendReceive`	netcoreapp3.1	1.84μs	1.66ns	6.42ns	0.0147	1.1 KB
master	`SendReceive`	net472	2.19μs	2.87ns	10.7ns	0.177	1.12 KB
#5031	`SendReceive`	net6.0	1.31μs	1.54ns	5.95ns	0.0157	1.1 KB
#5031	`SendReceive`	netcoreapp3.1	1.71μs	0.387ns	1.34ns	0.0149	1.1 KB
#5031	`SendReceive`	net472	2.13μs	1.35ns	4.86ns	0.177	1.12 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.82μs	0.894ns	3.46ns	0.0211	1.53 KB
master	`EnrichedLog`	netcoreapp3.1	3.91μs	1.2ns	4.5ns	0.0198	1.58 KB
master	`EnrichedLog`	net472	4.31μs	0.795ns	3.08ns	0.311	1.97 KB
#5031	`EnrichedLog`	net6.0	2.9μs	1.04ns	4.03ns	0.0215	1.53 KB
#5031	`EnrichedLog`	netcoreapp3.1	3.92μs	2.14ns	8.28ns	0.0214	1.58 KB
#5031	`EnrichedLog`	net472	4.21μs	2.18ns	8.43ns	0.311	1.97 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5031

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.176	548.98	645.71

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	449ns	0.101ns	0.379ns	0.00744	536 B
master	`StartFinishSpan`	netcoreapp3.1	750ns	0.148ns	0.554ns	0.00701	536 B
master	`StartFinishSpan`	net472	707ns	0.155ns	0.579ns	0.0852	538 B
master	`StartFinishScope`	net6.0	549ns	0.247ns	0.923ns	0.00922	656 B
master	`StartFinishScope`	netcoreapp3.1	815ns	0.411ns	1.59ns	0.00869	656 B
master	`StartFinishScope`	net472	993ns	0.539ns	2.09ns	0.0978	618 B
#5031	`StartFinishSpan`	net6.0	470ns	0.207ns	0.745ns	0.00757	536 B
#5031	`StartFinishSpan`	netcoreapp3.1	746ns	2.8ns	10.8ns	0.00708	536 B
#5031	`StartFinishSpan`	net472	782ns	0.782ns	3.03ns	0.0853	538 B
#5031	`StartFinishScope`	net6.0	646ns	0.396ns	1.53ns	0.00915	656 B
#5031	`StartFinishScope`	netcoreapp3.1	813ns	0.489ns	1.9ns	0.00884	656 B
#5031	`StartFinishScope`	net472	978ns	0.372ns	1.39ns	0.098	618 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5031

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.157	607.75	702.87

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	608ns	0.476ns	1.84ns	0.00926	656 B
master	`RunOnMethodBegin`	netcoreapp3.1	982ns	0.31ns	1.12ns	0.00863	656 B
master	`RunOnMethodBegin`	net472	1.11μs	0.391ns	1.51ns	0.098	618 B
#5031	`RunOnMethodBegin`	net6.0	703ns	0.243ns	0.939ns	0.00914	656 B
#5031	`RunOnMethodBegin`	netcoreapp3.1	986ns	0.498ns	1.93ns	0.00882	656 B
#5031	`RunOnMethodBegin`	net472	1.09μs	0.212ns	0.765ns	0.0979	618 B

andrewlock · 2024-01-11T17:04:05Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5031) (11.352M)   : 0, 11351789
    master (11.632M)   : 0, 11631618
    benchmarks/2.9.0 (11.757M)   : 0, 11756801

    section Automatic
    This PR (5031) (7.941M)   : 0, 7941299
    master (8.011M)   : 0, 8010820
    benchmarks/2.9.0 (8.573M)   : 0, 8573327

    section Trace stats
    This PR (5031) (8.405M)   : 0, 8405138
    master (8.335M)   : 0, 8335191

    section Manual
    This PR (5031) (9.896M)   : 0, 9896476
    master (10.240M)   : 0, 10240464

    section Manual + Automatic
    This PR (5031) (7.453M)   : 0, 7452617
    master (7.615M)   : 0, 7614917

    section Version Conflict
    This PR (5031) (6.752M)   : 0, 6751801
    master (6.860M)   : 0, 6859931

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5031) (9.689M)   : 0, 9689028
    master (9.594M)   : 0, 9594448
    benchmarks/2.9.0 (9.605M)   : 0, 9605360

    section Automatic
    This PR (5031) (6.649M)   : 0, 6648901
    master (6.570M)   : 0, 6569521

    section Trace stats
    This PR (5031) (6.762M)   : 0, 6762365
    master (6.865M)   : 0, 6865257

    section Manual
    This PR (5031) (8.354M)   : 0, 8353704
    master (8.346M)   : 0, 8345546

    section Manual + Automatic
    This PR (5031) (6.294M)   : 0, 6293761
    master (6.111M)   : 0, 6110796

    section Version Conflict
    This PR (5031) (5.766M)   : 0, 5766328
    master (5.680M)   : 0, 5679792

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5031) (9.982M)   : 0, 9981620
    master (10.186M)   : 0, 10186222
    benchmarks/2.9.0 (9.956M)   : 0, 9956027

    section Automatic
    This PR (5031) (7.077M)   : 0, 7077078
    master (7.005M)   : 0, 7005346
    benchmarks/2.9.0 (7.438M)   : 0, 7437608

    section Trace stats
    This PR (5031) (7.476M)   : 0, 7476250
    master (7.461M)   : 0, 7461480

    section Manual
    This PR (5031) (8.790M)   : 0, 8790144
    master (8.893M)   : 0, 8893402

    section Manual + Automatic
    This PR (5031) (6.818M)   : 0, 6817532
    master (6.733M)   : 0, 6733115

    section Version Conflict
    This PR (5031) (6.158M)   : 0, 6158028
    master (6.183M)   : 0, 6183301

gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.421M)   : 0, 7421031
    benchmarks/2.9.0 (7.813M)   : 0, 7813475

    section No attack
    master (1.758M)   : 0, 1758315
    benchmarks/2.9.0 (3.251M)   : 0, 3251456

    section Attack
    master (1.406M)   : 0, 1405570
    benchmarks/2.9.0 (2.577M)   : 0, 2577462

    section Blocking
    master (3.114M)   : 0, 3114389

    section IAST default
    master (6.585M)   : 0, 6584825

    section IAST full
    master (5.824M)   : 0, 5824226

    section Base vuln
    master (0.971M)   : 0, 971422

    section IAST vuln
    master (0.888M)   : 0, 888441

andrewlock

I'd be inclined to leave it as just this one for now, and see if there's any impact on CI, before we go all in

andrewlock · 2024-01-12T17:15:24Z

...Datadog.Trace.ClrProfiler.IntegrationTests/Datadog.Trace.ClrProfiler.IntegrationTests.csproj

@@ -27,6 +27,8 @@

    <PackageReference Include="System.Collections.Immutable" Version="5.0.0" />

+    <PackageReference Include="Testcontainers" Version="3.6.0" />
+    <PackageReference Include="StrongNamer" Version="0.2.5" />  


grmbl grmbl ihatestrongnaming grmbl grmbl

tracer/test/Datadog.Trace.TestHelpers.AutoInstrumentation/Containers/AerospikeFixture.cs

tracer/test/Datadog.Trace.TestHelpers.AutoInstrumentation/Containers/ContainerFixture.cs

tracer/test/Datadog.Trace.TestHelpers.AutoInstrumentation/DockerTestFramework.cs

andrewlock · 2024-01-12T17:21:50Z

tracer/test/Datadog.Trace.TestHelpers.AutoInstrumentation/DockerTestFramework.cs

+                            var fixture = (ContainerFixture)Activator.CreateInstance(fixtureType);
+                            await fixture!.InitializeAsync();


Out of interest, why do we need this? 🤔 Doesn't xunit automatically call this for fixtures?

We don't need this. But I prefer to give the responsibility of starting the containers to the executor, at a well-defined time, rather than letting xunit do its thing.

…ainers/AerospikeFixture.cs Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>

…ainers/ContainerFixture.cs Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>

datadog-ddstaging · 2025-01-13T12:41:20Z

Datadog Report

Branch report: kevin/test-containers-mvp
Commit report: d2f13ab
Test service: dd-trace-dotnet

✅ 0 Failed, 232357 Passed, 1940 Skipped, 18h 59m 14.52s Total Time
❄️ 1 New Flaky

New Flaky Tests (1)

EarlyFlakeDetection - Datadog.Trace.ClrProfiler.IntegrationTests.CI.MsTestV2EvpTests - Last Failure
Expand for error
```
 The sample did not exit in 600000ms. Memory dump taken: True. Killing process.
```

andrewlock · 2025-01-13T15:14:57Z

Benchmarks Report for tracer 🐌

Benchmarks for #5031 compared to master:

3 benchmarks are faster, with geometric mean 1.123
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.01μs	44.9ns	280ns	0.015	0.0075	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.3μs	52.5ns	246ns	0.0208	0.0104	0	5.8 KB
master	`StartStopWithChild`	net472	16.3μs	31.3ns	121ns	1.03	0.296	0.0961	6.22 KB
#5031	`StartStopWithChild`	net6.0	7.88μs	42.6ns	252ns	0.0154	0.00771	0	5.61 KB
#5031	`StartStopWithChild`	netcoreapp3.1	10.4μs	55.3ns	287ns	0.0253	0.0101	0	5.8 KB
#5031	`StartStopWithChild`	net472	16.5μs	48.3ns	187ns	1.04	0.317	0.0977	6.2 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	503μs	219ns	821ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	635μs	277ns	1.03μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	856μs	374ns	1.4μs	0.428	3.3 KB
#5031	`WriteAndFlushEnrichedTraces`	net6.0	491μs	130ns	487ns	0	2.7 KB
#5031	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	661μs	413ns	1.54μs	0	2.7 KB
#5031	`WriteAndFlushEnrichedTraces`	net472	855μs	546ns	2.11μs	0.425	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	154μs	900ns	7.85μs	0.18	14.47 KB
master	`SendRequest`	netcoreapp3.1	176μs	1.13μs	11.2μs	0.17	17.27 KB
master	`SendRequest`	net472	0.00287ns	0.00089ns	0.00345ns	0	0 b
#5031	`SendRequest`	net6.0	146μs	820ns	6.91μs	0.14	14.47 KB
#5031	`SendRequest`	netcoreapp3.1	176μs	1.14μs	11.4μs	0.177	17.27 KB
#5031	`SendRequest`	net472	0.000299ns	0.000299ns	0.00108ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	588μs	1.98μs	7.15μs	0.604	0	0	41.72 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	678μs	3.54μs	21.8μs	0.316	0	0	41.75 KB
master	`WriteAndFlushEnrichedTraces`	net472	842μs	3.75μs	14μs	8.62	2.59	0.431	53.31 KB
#5031	`WriteAndFlushEnrichedTraces`	net6.0	562μs	3.05μs	17μs	0.581	0	0	41.64 KB
#5031	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	657μs	3.08μs	12.7μs	0.326	0	0	41.74 KB
#5031	`WriteAndFlushEnrichedTraces`	net472	825μs	4.12μs	17.5μs	8.39	2.52	0.419	53.24 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.4μs	1.78ns	6.89ns	0.014	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.85μs	1.1ns	4.11ns	0.0139	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.08μs	1.71ns	6.4ns	0.156	0.00103	987 B
#5031	`ExecuteNonQuery`	net6.0	1.36μs	1.62ns	6.27ns	0.0143	0	1.02 KB
#5031	`ExecuteNonQuery`	netcoreapp3.1	1.76μs	1.2ns	4.63ns	0.0132	0	1.02 KB
#5031	`ExecuteNonQuery`	net472	2.15μs	0.866ns	3ns	0.156	0.00107	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.24μs	0.817ns	3.17ns	0.0137	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.57μs	0.726ns	2.71ns	0.0134	976 B
master	`CallElasticsearch`	net472	2.55μs	1.53ns	5.71ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.29μs	0.754ns	2.92ns	0.0129	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.63μs	0.978ns	3.79ns	0.0139	1.02 KB
master	`CallElasticsearchAsync`	net472	2.57μs	1.83ns	7.08ns	0.167	1.05 KB
#5031	`CallElasticsearch`	net6.0	1.24μs	0.614ns	2.38ns	0.0136	976 B
#5031	`CallElasticsearch`	netcoreapp3.1	1.64μs	1.24ns	4.81ns	0.0131	976 B
#5031	`CallElasticsearch`	net472	2.55μs	1.49ns	5.37ns	0.157	995 B
#5031	`CallElasticsearchAsync`	net6.0	1.33μs	0.782ns	3.03ns	0.0134	952 B
#5031	`CallElasticsearchAsync`	netcoreapp3.1	1.69μs	1.34ns	5.19ns	0.0136	1.02 KB
#5031	`CallElasticsearchAsync`	net472	2.76μs	2ns	7.76ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5031

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net472	1.119	1,928.82	1,724.13

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.27μs	1.16ns	4.34ns	0.0134	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.53μs	1.64ns	5.91ns	0.0133	952 B
master	`ExecuteAsync`	net472	1.93μs	0.401ns	1.39ns	0.144	915 B
#5031	`ExecuteAsync`	net6.0	1.32μs	1.04ns	4.03ns	0.0132	952 B
#5031	`ExecuteAsync`	netcoreapp3.1	1.68μs	0.968ns	3.62ns	0.0126	952 B
#5031	`ExecuteAsync`	net472	1.72μs	0.526ns	1.97ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.38μs	2.19ns	8.5ns	0.031	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.45μs	1.83ns	6.85ns	0.0379	2.85 KB
master	`SendAsync`	net472	7.39μs	3.2ns	12.4ns	0.492	3.12 KB
#5031	`SendAsync`	net6.0	4.43μs	1.11ns	4.15ns	0.0332	2.31 KB
#5031	`SendAsync`	netcoreapp3.1	5.37μs	2.41ns	9.34ns	0.0377	2.85 KB
#5031	`SendAsync`	net472	7.4μs	2.44ns	9.45ns	0.496	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.52μs	0.687ns	2.66ns	0.023	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.21μs	0.989ns	3.57ns	0.0225	1.64 KB
master	`EnrichedLog`	net472	2.86μs	1.97ns	7.63ns	0.249	1.57 KB
#5031	`EnrichedLog`	net6.0	1.45μs	0.732ns	2.74ns	0.0231	1.64 KB
#5031	`EnrichedLog`	netcoreapp3.1	2.17μs	1.03ns	3.72ns	0.0219	1.64 KB
#5031	`EnrichedLog`	net472	2.7μs	1.61ns	6.23ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	118μs	258ns	932ns	0	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	121μs	232ns	900ns	0.0613	0	4.28 KB
master	`EnrichedLog`	net472	152μs	118ns	458ns	0.683	0.228	4.46 KB
#5031	`EnrichedLog`	net6.0	118μs	113ns	437ns	0.0589	0	4.28 KB
#5031	`EnrichedLog`	netcoreapp3.1	123μs	174ns	674ns	0	0	4.28 KB
#5031	`EnrichedLog`	net472	152μs	162ns	628ns	0.688	0.229	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.97μs	15.8ns	79ns	0.0304	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.18μs	1.68ns	6.28ns	0.0292	2.2 KB
master	`EnrichedLog`	net472	5.18μs	1.24ns	4.8ns	0.319	2.02 KB
#5031	`EnrichedLog`	net6.0	3.03μs	1.61ns	6.24ns	0.0305	2.2 KB
#5031	`EnrichedLog`	netcoreapp3.1	4.29μs	2.47ns	9.26ns	0.0277	2.2 KB
#5031	`EnrichedLog`	net472	4.9μs	1.23ns	4.43ns	0.32	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.35μs	0.594ns	2.3ns	0.0161	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.68μs	0.561ns	2.17ns	0.0149	1.14 KB
master	`SendReceive`	net472	2.14μs	0.793ns	3.07ns	0.183	1.16 KB
#5031	`SendReceive`	net6.0	1.36μs	0.627ns	2.43ns	0.0162	1.14 KB
#5031	`SendReceive`	netcoreapp3.1	1.78μs	2.79ns	10.8ns	0.0151	1.14 KB
#5031	`SendReceive`	net472	2.14μs	1.53ns	5.91ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.73μs	0.698ns	2.7ns	0.0219	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	4.1μs	1.84ns	7.13ns	0.0226	1.65 KB
master	`EnrichedLog`	net472	4.34μs	1.98ns	7.15ns	0.322	2.04 KB
#5031	`EnrichedLog`	net6.0	2.83μs	3.5ns	13.5ns	0.0227	1.6 KB
#5031	`EnrichedLog`	netcoreapp3.1	3.95μs	1.27ns	4.94ns	0.0217	1.65 KB
#5031	`EnrichedLog`	net472	4.43μs	3.06ns	11.9ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5031

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472	1.129	943.60	835.42
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0	1.120	451.78	403.42

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	452ns	0.246ns	0.951ns	0.00818	576 B
master	`StartFinishSpan`	netcoreapp3.1	615ns	1.03ns	3.98ns	0.00804	576 B
master	`StartFinishSpan`	net472	672ns	0.322ns	1.25ns	0.0916	578 B
master	`StartFinishScope`	net6.0	569ns	0.439ns	1.7ns	0.0097	696 B
master	`StartFinishScope`	netcoreapp3.1	720ns	0.908ns	3.52ns	0.00955	696 B
master	`StartFinishScope`	net472	943ns	0.769ns	2.88ns	0.104	658 B
#5031	`StartFinishSpan`	net6.0	403ns	0.224ns	0.868ns	0.00811	576 B
#5031	`StartFinishSpan`	netcoreapp3.1	574ns	0.593ns	2.3ns	0.00776	576 B
#5031	`StartFinishSpan`	net472	708ns	0.524ns	2.03ns	0.0917	578 B
#5031	`StartFinishScope`	net6.0	558ns	0.393ns	1.52ns	0.00974	696 B
#5031	`StartFinishScope`	netcoreapp3.1	694ns	0.605ns	2.1ns	0.00946	696 B
#5031	`StartFinishScope`	net472	835ns	0.454ns	1.7ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	659ns	0.43ns	1.67ns	0.00961	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	934ns	0.414ns	1.43ns	0.00906	696 B
master	`RunOnMethodBegin`	net472	1.04μs	0.671ns	2.42ns	0.104	658 B
#5031	`RunOnMethodBegin`	net6.0	662ns	0.382ns	1.48ns	0.0096	696 B
#5031	`RunOnMethodBegin`	netcoreapp3.1	926ns	1.08ns	4.17ns	0.00913	696 B
#5031	`RunOnMethodBegin`	net472	1.11μs	0.863ns	3.23ns	0.104	658 B

kevingosse force-pushed the kevin/test-containers-mvp branch 2 times, most recently from 5489c82 to f49fd95 Compare January 9, 2024 14:34

andrewlock approved these changes Jan 12, 2024

View reviewed changes

kevingosse marked this pull request as ready for review January 15, 2024 11:58

kevingosse requested review from a team as code owners January 15, 2024 11:58

kevingosse force-pushed the kevin/test-containers-mvp branch 2 times, most recently from eee782d to 8453bb5 Compare January 18, 2024 12:53

kevingosse force-pushed the kevin/test-containers-mvp branch from 8453bb5 to 55f2ea0 Compare January 29, 2024 10:34

kevingosse and others added 13 commits January 13, 2025 12:14

MVP for test containers (with aerospike)

4b9c079

Try fixing the logger error

b0f21cb

Fix compilation error

2d70458

Move the testcontainers dependency

3503528

Fix compilation errors

c7f627c

Getting there...

05b1d8c

Set the stuff needed for wormholes (or something)

bc7c7fc

Try flipping switches

e4b5b56

Try connecting to localhost instead

08ef053

Try more stuff

56c5e9b

Update tracer/test/Datadog.Trace.TestHelpers.AutoInstrumentation/Cont…

4aaa8bb

…ainers/AerospikeFixture.cs Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>

Update tracer/test/Datadog.Trace.TestHelpers.AutoInstrumentation/Cont…

5ea69cb

…ainers/ContainerFixture.cs Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>

Refactor DockerTestFramework

a1dacec

andrewlock force-pushed the kevin/test-containers-mvp branch from 55f2ea0 to a1dacec Compare January 13, 2025 12:15

andrewlock requested a review from a team as a code owner January 13, 2025 12:15

Fix rebase issue

d2f13ab

andrewlock merged commit 2246c2d into master Jan 15, 2025
124 of 127 checks passed

andrewlock deleted the kevin/test-containers-mvp branch January 15, 2025 10:42

github-actions bot added this to the vNext-v3 milestone Jan 15, 2025

andrewlock added the area:tests unit tests, integration tests label Jan 22, 2025

		var fixture = (ContainerFixture)Activator.CreateInstance(fixtureType);
		await fixture!.InitializeAsync();

MVP for test containers (with aerospike) #5031

MVP for test containers (with aerospike) #5031

Conversation

kevingosse commented Jan 8, 2024 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

datadog-ddstaging bot commented Jan 8, 2024 • edited Loading

Datadog Report

andrewlock commented Jan 8, 2024 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Jan 8, 2024 • edited Loading

Benchmarks Report 🐌

Benchmark details

Raw results

Raw results

Faster 🎉 in #5031

Raw results

Raw results

Raw results

More allocations ⚠️ in #5031

Raw results

Raw results

Raw results

Raw results

Raw results

Fewer allocations 🎉 in #5031

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5031

Raw results

Slower ⚠️ in #5031

Raw results

andrewlock commented Jan 11, 2024 • edited Loading

Throughput/Crank Report:zap:

andrewlock left a comment

Choose a reason for hiding this comment

andrewlock Jan 12, 2024

Choose a reason for hiding this comment

kevingosse Jan 15, 2024

Choose a reason for hiding this comment

andrewlock Jan 12, 2024

Choose a reason for hiding this comment

kevingosse Jan 15, 2024

Choose a reason for hiding this comment

datadog-ddstaging bot commented Jan 13, 2025 • edited Loading

Datadog Report

New Flaky Tests (1)

andrewlock commented Jan 13, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #5031

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #5031

Raw results

Raw results

kevingosse commented Jan 8, 2024 •

edited

Loading

datadog-ddstaging bot commented Jan 8, 2024 •

edited

Loading

andrewlock commented Jan 8, 2024 •

edited

Loading

andrewlock commented Jan 8, 2024 •

edited

Loading

andrewlock commented Jan 11, 2024 •

edited

Loading

datadog-ddstaging bot commented Jan 13, 2025 •

edited

Loading