Optimize runtime metrics #5304

kevingosse · 2024-03-12T16:12:33Z

Summary of changes

Optimize the process information collection in the runtime metrics, on Windows.

Method	Mean	Error	StdDev	Ratio	Allocated	Alloc Ratio
Old	5,629.8 us	95.00 us	88.87 us	1.00	6765 B	1.00
New	246.3 us	2.49 us	2.21 us	0.04	-	0.00

Reason for change

We have seen multiple cases where it appeared as a hotspot in customers who enabled the continuous profiler (I'm not saying that the CP is the cause, just that we're blind for other customers). Also, I often see dead ProcessThread objects in memory dumps, indicating that it's a significant source of allocations.

Implementation details

The old version uses the .NET Process object to retrieve the required information. On Windows, it relies on NtQuerySystemInformation, which is very slow because it actually queries the information for all processes on the machine. To make things worse, .NET allocates ProcessThread objects for each thread, even though we're only interested in the count. To make worse things worse, ProcessThread objects have a finalizer, increasing the probability that they end up in gen 2.

This PR relies on the process snapshotting APIs introduced in Windows 8.1. They're much faster because they target a single process, and use copy-on-write. The old API is kept as a fallback in case we encounter a non-transient error (arbitrarily defined as "3 consecutive errors") or if the target application uses an older version of Windows.

I had to declare separate PSS_PROCESS_INFORMATION_32 and PSS_PROCESS_INFORMATION_64 structures because to my surprise the Pack argument of StructLayout is 8 even on 32 bit (I naively thought it used the pointer size). It doesn't look like there is any good solution for this today, except duplicating the code: dotnet/runtime#60573.

Test coverage

Added a unit test for the process metrics. They do only basic validation, as anything more complex would likely flake.

zacharycmontoya · 2024-03-12T16:51:42Z

tracer/src/Datadog.Trace/RuntimeMetrics/RuntimeMetricsWriter.cs

+
+                    if (_pssConsecutiveFailures >= 3)
+                    {
+                        Log.Error("Pss failed 3 times in a row, falling back to the Process API");


Could this be a Warning message instead?

I made it an error because it's not supposed to ever happen, and I'd like to see it in telemetry logs if it does.

datadog-ddstaging · 2024-03-12T16:54:25Z

Datadog Report

Branch report: kevin/thread_count
Commit report: bcc26b0
Test service: dd-trace-dotnet

✅ 0 Failed, 330087 Passed, 1597 Skipped, 35m 15.86s Wall Time

zacharycmontoya · 2024-03-12T17:06:15Z

tracer/src/Datadog.Trace/RuntimeMetrics/PssRuntimeInformation.cs

+        public long CreateTime;
+        public long ExitTime;
+        public long KernelTime;
+        public long UserTime;


Since the FILETIME structure is composed of two DWORDS (which I believe are unsigned 32-bit values), should this be a ulong value instead of a long value?

Ultimately I don't believe it will change anything because we give them to TimeSpan.FromTicks which expects a long (so we have to cast them back). Still, I suppose it's better to get the marshalled structure right.

Done in 0fe295c

zacharycmontoya · 2024-03-12T17:14:27Z

tracer/src/Datadog.Trace/RuntimeMetrics/PssRuntimeInformation.cs

+namespace Datadog.Trace.RuntimeMetrics;
+
+// ReSharper disable InconsistentNaming UnusedMember.Local
+internal class PssRuntimeInformation


Super small nit: Can we expand Pss to ProcessSnapshot in the class name, so ProcessSnapshotRuntimeInformation?

Done in 0fe295c

zacharycmontoya

Overall, LGTM! Nice work!

zacharycmontoya · 2024-03-12T17:52:40Z

Regarding the app trimming and assembly reference snapshot diffs, this PR adds a reference to the System.ComponentModel.Win32Exception type, which is in the Microsoft.Win32.Primitives. This new reference is acceptable because this is always included in the runtime ✅ The expected files now just need to be updated appropriately

andrewlock · 2024-03-12T19:54:54Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5304) - mean (71ms)  : 69, 73
     .   : milestone, 71,
    master - mean (73ms)  : 65, 82
     .   : milestone, 73,

    section CallTarget+Inlining+NGEN
    This PR (5304) - mean (1,008ms)  : 991, 1024
     .   : milestone, 1008,
    master - mean (1,005ms)  : 978, 1033
     .   : milestone, 1005,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5304) - mean (111ms)  : 108, 115
     .   : milestone, 111,
    master - mean (111ms)  : 108, 114
     .   : milestone, 111,

    section CallTarget+Inlining+NGEN
    This PR (5304) - mean (739ms)  : 719, 759
     .   : milestone, 739,
    master - mean (733ms)  : 711, 754
     .   : milestone, 733,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5304) - mean (94ms)  : 91, 96
     .   : milestone, 94,
    master - mean (93ms)  : 91, 96
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (5304) - mean (686ms)  : 660, 711
     .   : milestone, 686,
    master - mean (683ms)  : 665, 702
     .   : milestone, 683,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5304) - mean (187ms)  : 184, 191
     .   : milestone, 187,
    master - mean (187ms)  : 185, 190
     .   : milestone, 187,

    section CallTarget+Inlining+NGEN
    This PR (5304) - mean (1,075ms)  : 1049, 1100
     .   : milestone, 1075,
    master - mean (1,079ms)  : 1060, 1099
     .   : milestone, 1079,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5304) - mean (271ms)  : 268, 275
     .   : milestone, 271,
    master - mean (271ms)  : 267, 275
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (5304) - mean (882ms)  : 859, 906
     .   : milestone, 882,
    master - mean (883ms)  : 861, 904
     .   : milestone, 883,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5304) - mean (260ms)  : 256, 264
     .   : milestone, 260,
    master - mean (260ms)  : 256, 264
     .   : milestone, 260,

    section CallTarget+Inlining+NGEN
    This PR (5304) - mean (858ms)  : 830, 886
     .   : milestone, 858,
    master - mean (861ms)  : 841, 882
     .   : milestone, 861,

andrewlock · 2024-03-13T01:12:43Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5304) (11.100M)   : 0, 11099977
    master (11.029M)   : 0, 11029393
    benchmarks/2.9.0 (11.199M)   : 0, 11198849

    section Automatic
    This PR (5304) (7.533M)   : 0, 7532883
    master (7.725M)   : 0, 7725218
    benchmarks/2.9.0 (8.073M)   : 0, 8073104

    section Trace stats
    This PR (5304) (7.949M)   : 0, 7948656
    master (7.999M)   : 0, 7999147

    section Manual
    This PR (5304) (9.574M)   : 0, 9574277
    master (9.720M)   : 0, 9720192

    section Manual + Automatic
    This PR (5304) (7.234M)   : 0, 7233944
    master (7.300M)   : 0, 7299991

    section Version Conflict
    This PR (5304) (6.502M)   : 0, 6501952
    master (6.535M)   : 0, 6534605

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5304) (9.451M)   : 0, 9451208
    master (9.645M)   : 0, 9645161
    benchmarks/2.9.0 (9.424M)   : 0, 9424414

    section Automatic
    This PR (5304) (6.661M)   : 0, 6660745
    master (6.570M)   : 0, 6569649

    section Trace stats
    This PR (5304) (6.797M)   : 0, 6796817
    master (6.920M)   : 0, 6919843

    section Manual
    This PR (5304) (8.324M)   : 0, 8323730
    master (8.279M)   : 0, 8278546

    section Manual + Automatic
    This PR (5304) (6.232M)   : 0, 6231846
    master (6.119M)   : 0, 6118527

    section Version Conflict
    This PR (5304) (5.513M)   : 0, 5512778
    master (5.557M)   : 0, 5557390

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5304) (10.072M)   : 0, 10071694
    master (9.952M)   : 0, 9952258
    benchmarks/2.9.0 (9.559M)   : 0, 9558711

    section Automatic
    This PR (5304) (7.191M)   : 0, 7190754
    master (6.660M)   : 0, 6659881
    benchmarks/2.9.0 (7.049M)   : 0, 7049098

    section Trace stats
    This PR (5304) (7.388M)   : 0, 7387714
    master (7.461M)   : 0, 7460529

    section Manual
    This PR (5304) (8.764M)   : 0, 8764246
    master (8.877M)   : 0, 8876838

    section Manual + Automatic
    This PR (5304) (6.855M)   : 0, 6854979
    master (6.824M)   : 0, 6823787

    section Version Conflict
    This PR (5304) (6.262M)   : 0, 6262132
    master (6.200M)   : 0, 6199775

andrewlock

LGTM asfar as I can tell, but I'm trusting you that the structs are all correct etc :D

tracer/src/Datadog.Trace/RuntimeMetrics/ProcessSnapshotRuntimeInformation.cs

andrewlock · 2024-03-13T09:56:56Z

tracer/src/Datadog.Trace/RuntimeMetrics/ProcessSnapshotRuntimeInformation.cs

+
+                if (result != 0)
+                {
+                    Log.Error<IntPtr, int>("PssFreeSnapshot returned an error, the tracer might be leaking memory. Handle: {Handle}. Error code: {Result}.", snapshotHandle, result);


andrewlock · 2024-03-13T12:11:13Z

tracer/src/Datadog.Trace/RuntimeMetrics/RuntimeMetricsWriter.cs

+                }
+                catch
+                {
+                    _pssConsecutiveFailures += 1;


We can technically have more than one RuntimeMetricsWriter in version conflict, right? Should we use interlocked here?

But then they're different RuntimeMetricsWriter and they all have their own counter

True. And we never have more than one instance of the same type, when we replace the tracer instance for example (before the old one has been disposed)?

Though honestly I can't think of any reason not to use an interlocked here so I guess I can play safe and add one

…Information.cs Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>

andrewlock · 2024-03-13T18:45:18Z

Benchmarks Report for tracer 🐌

Benchmarks for #5304 compared to master:

2 benchmarks are slower, with geometric mean 1.166
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.69μs	48.5ns	307ns	0.022	0.0088	0	7.5 KB
master	`StartStopWithChild`	netcoreapp3.1	10.7μs	56.7ns	350ns	0.0322	0.0107	0.00537	7.6 KB
master	`StartStopWithChild`	net472	17.1μs	56.5ns	219ns	1.37	0.357	0.128	8.12 KB
#5304	`StartStopWithChild`	net6.0	8.47μs	46.5ns	275ns	0.024	0.012	0	7.5 KB
#5304	`StartStopWithChild`	netcoreapp3.1	10.5μs	58ns	343ns	0.0199	0.00997	0	7.6 KB
#5304	`StartStopWithChild`	net472	17μs	55.2ns	207ns	1.38	0.377	0.126	8.14 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	457μs	178ns	642ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	598μs	177ns	661ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	814μs	287ns	1.03μs	0.437	3.3 KB
#5304	`WriteAndFlushEnrichedTraces`	net6.0	452μs	349ns	1.35μs	0	2.7 KB
#5304	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	599μs	210ns	727ns	0	2.7 KB
#5304	`WriteAndFlushEnrichedTraces`	net472	813μs	278ns	1.08μs	0.403	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	170μs	109ns	422ns	0.17	18.33 KB
master	`SendRequest`	netcoreapp3.1	191μs	494ns	1.91μs	0.19	20.49 KB
master	`SendRequest`	net472	0.00136ns	0.000452ns	0.00169ns	0	0 b
#5304	`SendRequest`	net6.0	170μs	113ns	437ns	0.253	18.33 KB
#5304	`SendRequest`	netcoreapp3.1	192μs	127ns	474ns	0.192	20.49 KB
#5304	`SendRequest`	net472	5.86E‑05ns	3.22E‑05ns	0.000121ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	545μs	792ns	3.07μs	0.558	0	0	41.66 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	663μs	877ns	3.28μs	0.326	0	0	41.76 KB
master	`WriteAndFlushEnrichedTraces`	net472	840μs	3.51μs	13.6μs	8.33	2.5	0.417	53.23 KB
#5304	`WriteAndFlushEnrichedTraces`	net6.0	536μs	305ns	1.14μs	0.536	0	0	41.54 KB
#5304	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	657μs	466ns	1.8μs	0.332	0	0	41.77 KB
#5304	`WriteAndFlushEnrichedTraces`	net472	850μs	3.52μs	13.6μs	8.3	2.62	0.437	53.34 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.1μs	0.453ns	1.76ns	0.0111	784 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.55μs	0.568ns	2.2ns	0.0102	784 B
master	`ExecuteNonQuery`	net472	1.87μs	1.76ns	6.81ns	0.118	746 B
#5304	`ExecuteNonQuery`	net6.0	1.19μs	1.12ns	4.33ns	0.0108	784 B
#5304	`ExecuteNonQuery`	netcoreapp3.1	1.47μs	1.27ns	4.91ns	0.0102	784 B
#5304	`ExecuteNonQuery`	net472	1.76μs	0.761ns	2.95ns	0.118	746 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.31μs	0.615ns	2.22ns	0.0144	1.01 KB
master	`CallElasticsearch`	netcoreapp3.1	1.64μs	1.21ns	4.51ns	0.0139	1.01 KB
master	`CallElasticsearch`	net472	2.45μs	1.26ns	4.55ns	0.161	1.02 KB
master	`CallElasticsearchAsync`	net6.0	1.4μs	0.753ns	2.82ns	0.014	984 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.6μs	0.868ns	3.25ns	0.0145	1.06 KB
master	`CallElasticsearchAsync`	net472	2.53μs	1.19ns	4.6ns	0.17	1.08 KB
#5304	`CallElasticsearch`	net6.0	1.27μs	0.653ns	2.53ns	0.014	1.01 KB
#5304	`CallElasticsearch`	netcoreapp3.1	1.6μs	0.808ns	3.02ns	0.0137	1.01 KB
#5304	`CallElasticsearch`	net472	2.46μs	1.21ns	4.52ns	0.162	1.02 KB
#5304	`CallElasticsearchAsync`	net6.0	1.34μs	0.481ns	1.86ns	0.0141	984 B
#5304	`CallElasticsearchAsync`	netcoreapp3.1	1.67μs	1.57ns	6.07ns	0.0144	1.06 KB
#5304	`CallElasticsearchAsync`	net472	2.61μs	1.57ns	6.08ns	0.17	1.08 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.22μs	1.26ns	4.88ns	0.0127	928 B
master	`ExecuteAsync`	netcoreapp3.1	1.69μs	0.938ns	3.63ns	0.0125	928 B
master	`ExecuteAsync`	net472	1.9μs	1.86ns	7.21ns	0.141	891 B
#5304	`ExecuteAsync`	net6.0	1.31μs	1.04ns	4.02ns	0.0132	928 B
#5304	`ExecuteAsync`	netcoreapp3.1	1.66μs	1.33ns	4.79ns	0.0124	928 B
#5304	`ExecuteAsync`	net472	1.88μs	0.728ns	2.82ns	0.141	891 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.18μs	2.03ns	7.59ns	0.0293	2.16 KB
master	`SendAsync`	netcoreapp3.1	5.08μs	2.91ns	10.9ns	0.0356	2.7 KB
master	`SendAsync`	net472	7.79μs	8.2ns	31.8ns	0.533	3.38 KB
#5304	`SendAsync`	net6.0	4.28μs	2.03ns	7.6ns	0.0297	2.16 KB
#5304	`SendAsync`	netcoreapp3.1	5.1μs	3.68ns	13.8ns	0.0356	2.7 KB
#5304	`SendAsync`	net472	7.79μs	4.08ns	15.3ns	0.536	3.38 KB

Benchmarks.Trace.ILoggerBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5304

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog‑net6.0	1.141	1,447.84	1,652.56

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.45μs	0.629ns	2.35ns	0.0231	1.65 KB
master	`EnrichedLog`	netcoreapp3.1	2.11μs	0.959ns	3.71ns	0.0221	1.65 KB
master	`EnrichedLog`	net472	2.54μs	1.16ns	4.34ns	0.25	1.57 KB
#5304	`EnrichedLog`	net6.0	1.65μs	0.571ns	2.21ns	0.023	1.65 KB
#5304	`EnrichedLog`	netcoreapp3.1	2.22μs	1.38ns	5.18ns	0.0225	1.65 KB
#5304	`EnrichedLog`	net472	2.55μs	1.2ns	4.48ns	0.25	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	113μs	197ns	763ns	0	0	4.23 KB
master	`EnrichedLog`	netcoreapp3.1	118μs	102ns	381ns	0.0592	0	4.23 KB
master	`EnrichedLog`	net472	145μs	92.8ns	360ns	0.654	0.218	4.41 KB
#5304	`EnrichedLog`	net6.0	114μs	48.8ns	183ns	0.0574	0	4.23 KB
#5304	`EnrichedLog`	netcoreapp3.1	118μs	124ns	479ns	0.0593	0	4.23 KB
#5304	`EnrichedLog`	net472	147μs	53.4ns	207ns	0.662	0.221	4.41 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3μs	1.16ns	4.03ns	0.0303	2.21 KB
master	`EnrichedLog`	netcoreapp3.1	4.28μs	2.4ns	8.97ns	0.0298	2.21 KB
master	`EnrichedLog`	net472	4.84μs	2.15ns	8.06ns	0.321	2.02 KB
#5304	`EnrichedLog`	net6.0	3.19μs	11.7ns	58.4ns	0.0311	2.21 KB
#5304	`EnrichedLog`	netcoreapp3.1	4.14μs	3.33ns	12.9ns	0.0287	2.21 KB
#5304	`EnrichedLog`	net472	4.97μs	1.09ns	4.09ns	0.32	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.41μs	0.571ns	2.21ns	0.0162	1.18 KB
master	`SendReceive`	netcoreapp3.1	1.87μs	1.14ns	4.27ns	0.0159	1.18 KB
master	`SendReceive`	net472	2.3μs	1ns	3.89ns	0.187	1.18 KB
#5304	`SendReceive`	net6.0	1.5μs	0.986ns	3.55ns	0.0165	1.18 KB
#5304	`SendReceive`	netcoreapp3.1	1.81μs	1.43ns	5.35ns	0.0163	1.18 KB
#5304	`SendReceive`	net472	2.23μs	0.704ns	2.63ns	0.186	1.18 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.69μs	0.905ns	3.39ns	0.0215	1.55 KB
master	`EnrichedLog`	netcoreapp3.1	3.9μs	1ns	3.89ns	0.0213	1.6 KB
master	`EnrichedLog`	net472	4.43μs	2.12ns	8.23ns	0.315	1.99 KB
#5304	`EnrichedLog`	net6.0	2.75μs	1.07ns	4.01ns	0.022	1.55 KB
#5304	`EnrichedLog`	netcoreapp3.1	4.1μs	15ns	58.1ns	0.0219	1.6 KB
#5304	`EnrichedLog`	net472	4.56μs	1.79ns	6.69ns	0.316	1.99 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5304

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0	1.191	451.45	537.69

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	451ns	0.527ns	1.97ns	0.00771	552 B
master	`StartFinishSpan`	netcoreapp3.1	682ns	1.07ns	4.15ns	0.00736	552 B
master	`StartFinishSpan`	net472	730ns	0.393ns	1.52ns	0.0877	554 B
master	`StartFinishScope`	net6.0	600ns	0.714ns	2.67ns	0.00952	672 B
master	`StartFinishScope`	netcoreapp3.1	813ns	1.04ns	4.04ns	0.0089	672 B
master	`StartFinishScope`	net472	1.01μs	1.03ns	3.86ns	0.1	634 B
#5304	`StartFinishSpan`	net6.0	537ns	0.331ns	1.28ns	0.00767	552 B
#5304	`StartFinishSpan`	netcoreapp3.1	700ns	1.22ns	4.41ns	0.00733	552 B
#5304	`StartFinishSpan`	net472	770ns	1.5ns	5.79ns	0.0878	554 B
#5304	`StartFinishScope`	net6.0	586ns	0.62ns	2.32ns	0.00946	672 B
#5304	`StartFinishScope`	netcoreapp3.1	848ns	1.14ns	4.41ns	0.00881	672 B
#5304	`StartFinishScope`	net472	954ns	1.68ns	6.52ns	0.1	634 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	700ns	0.762ns	2.95ns	0.00944	672 B
master	`RunOnMethodBegin`	netcoreapp3.1	993ns	1.81ns	7.01ns	0.00908	672 B
master	`RunOnMethodBegin`	net472	1.09μs	1.73ns	6.72ns	0.101	634 B
#5304	`RunOnMethodBegin`	net6.0	750ns	1.05ns	4.08ns	0.00948	672 B
#5304	`RunOnMethodBegin`	netcoreapp3.1	987ns	0.626ns	2.42ns	0.00922	672 B
#5304	`RunOnMethodBegin`	net472	1.04μs	1.79ns	6.93ns	0.1	634 B

Optimize runtime metrics

fc666f8

kevingosse requested a review from a team as a code owner March 12, 2024 16:12

Enable nullability annotations

395b7bb

zacharycmontoya reviewed Mar 12, 2024

View reviewed changes

zacharycmontoya approved these changes Mar 12, 2024

View reviewed changes

Address PR feedback and update snapshots

0fe295c

andrewlock approved these changes Mar 13, 2024

View reviewed changes

andrewlock reviewed Mar 13, 2024

View reviewed changes

Update trimming, try fixing tests

c42fe0b

kevingosse requested a review from a team as a code owner March 13, 2024 12:27

kevingosse and others added 3 commits March 13, 2024 13:55

Update tracer/src/Datadog.Trace/RuntimeMetrics/ProcessSnapshotRuntime…

3a30953

…Information.cs Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>

Bump the number of threads, make test skippable

edcff71

Force to use a bit of CPU

6d94c9b

Use Interlocked.Increment

bcc26b0

kevingosse merged commit 3dab836 into master Mar 15, 2024
56 checks passed

kevingosse deleted the kevin/thread_count branch March 15, 2024 12:07

github-actions bot added this to the vNext milestone Mar 15, 2024

link04 added area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) area:runtime-metrics labels Mar 18, 2024

bchavez mentioned this pull request Oct 21, 2024

Kernel32!Pss*Snapshot Thread Capture Counts are not the same as Active Thread Counts for runtime.dotnet.threads.count in DD Tracer v2.49+ #6172

Closed

Optimize runtime metrics #5304

Optimize runtime metrics #5304

Conversation

kevingosse commented Mar 12, 2024 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

Choose a reason for hiding this comment

kevingosse Mar 12, 2024 • edited Loading

Choose a reason for hiding this comment

datadog-ddstaging bot commented Mar 12, 2024 • edited Loading

Datadog Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zacharycmontoya left a comment

Choose a reason for hiding this comment

zacharycmontoya commented Mar 12, 2024

andrewlock commented Mar 12, 2024 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Mar 13, 2024 • edited Loading

Throughput/Crank Report:zap:

andrewlock left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewlock commented Mar 13, 2024 • edited Loading

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5304

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #5304

Raw results

Raw results

kevingosse commented Mar 12, 2024 •

edited

Loading

kevingosse Mar 12, 2024 •

edited

Loading

datadog-ddstaging bot commented Mar 12, 2024 •

edited

Loading

andrewlock commented Mar 12, 2024 •

edited

Loading

andrewlock commented Mar 13, 2024 •

edited

Loading

andrewlock commented Mar 13, 2024 •

edited

Loading