Add a Github action that ensures you correctly persist any changes to the samples #6536

andrewlock · 2025-01-13T13:24:08Z

Summary of changes

Adds a GitHub action that ensures you run RegenerateSolutions after adding a new sample to the main solution

Reason for change

In #6498, we now have a separate samples solution that we use to build the samples in a separate stage. However, that means when you add a new sample to Datadog.Trace.sln you should run the RegenerateSolutions target, otherwise you could get unexpected failures. This GitHub action ensures that you do this.

Implementation details

Added a github action check, same as we have for other similar issues

Test coverage

Can't test it until it's merged. Once I'm sure it's working correctly, will mark it as required

… the samples

datadog-ddstaging · 2025-01-13T15:11:47Z

Datadog Report

Branch report: andrew/ci/flag-missing-sample-build
Commit report: 5400260
Test service: dd-trace-dotnet

✅ 0 Failed, 240156 Passed, 1976 Skipped, 18h 39m 38.43s Total Time

andrewlock · 2025-01-13T15:55:05Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6536) - mean (69ms)  : 65, 73
     .   : milestone, 69,
    master - mean (68ms)  : 66, 71
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (6536) - mean (977ms)  : 944, 1010
     .   : milestone, 977,
    master - mean (979ms)  : 948, 1010
     .   : milestone, 979,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6536) - mean (107ms)  : 105, 109
     .   : milestone, 107,
    master - mean (107ms)  : 105, 109
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6536) - mean (673ms)  : 661, 685
     .   : milestone, 673,
    master - mean (678ms)  : 664, 693
     .   : milestone, 678,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6536) - mean (91ms)  : 88, 93
     .   : milestone, 91,
    master - mean (91ms)  : 89, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6536) - mean (630ms)  : 612, 648
     .   : milestone, 630,
    master - mean (635ms)  : 620, 649
     .   : milestone, 635,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6536) - mean (194ms)  : 186, 203
     .   : milestone, 194,
    master - mean (194ms)  : 190, 198
     .   : milestone, 194,

    section CallTarget+Inlining+NGEN
    This PR (6536) - mean (1,101ms)  : 1076, 1127
     .   : milestone, 1101,
    master - mean (1,103ms)  : 1066, 1140
     .   : milestone, 1103,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6536) - mean (279ms)  : 274, 284
     .   : milestone, 279,
    master - mean (277ms)  : 274, 281
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (6536) - mean (869ms)  : 844, 893
     .   : milestone, 869,
    master - mean (870ms)  : 843, 897
     .   : milestone, 870,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6536) - mean (267ms)  : 263, 270
     .   : milestone, 267,
    master - mean (267ms)  : 263, 271
     .   : milestone, 267,

    section CallTarget+Inlining+NGEN
    This PR (6536) - mean (846ms)  : 819, 874
     .   : milestone, 846,
    master - mean (852ms)  : 825, 878
     .   : milestone, 852,

andrewlock · 2025-01-13T16:43:05Z

Benchmarks Report for tracer 🐌

Benchmarks for #6536 compared to master:

1 benchmarks are faster, with geometric mean 1.164
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.01μs	44.9ns	280ns	0.015	0.0075	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.3μs	52.5ns	246ns	0.0208	0.0104	0	5.8 KB
master	`StartStopWithChild`	net472	16.3μs	31.3ns	121ns	1.03	0.296	0.0961	6.22 KB
#6536	`StartStopWithChild`	net6.0	8.32μs	46.9ns	335ns	0.0219	0.0131	0.00438	5.6 KB
#6536	`StartStopWithChild`	netcoreapp3.1	10.1μs	55.4ns	333ns	0.0157	0.00525	0	5.8 KB
#6536	`StartStopWithChild`	net472	16.5μs	58.2ns	210ns	1.05	0.317	0.106	6.22 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	503μs	219ns	821ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	635μs	277ns	1.03μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	856μs	374ns	1.4μs	0.428	3.3 KB
#6536	`WriteAndFlushEnrichedTraces`	net6.0	478μs	160ns	621ns	0	2.7 KB
#6536	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	651μs	689ns	2.67μs	0	2.7 KB
#6536	`WriteAndFlushEnrichedTraces`	net472	852μs	611ns	2.36μs	0.428	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	154μs	900ns	7.85μs	0.18	14.47 KB
master	`SendRequest`	netcoreapp3.1	176μs	1.13μs	11.2μs	0.17	17.27 KB
master	`SendRequest`	net472	0.00287ns	0.00089ns	0.00345ns	0	0 b
#6536	`SendRequest`	net6.0	152μs	891ns	8.45μs	0.151	14.47 KB
#6536	`SendRequest`	netcoreapp3.1	174μs	1.02μs	9.09μs	0.165	17.27 KB
#6536	`SendRequest`	net472	0.00669ns	0.0019ns	0.00736ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	588μs	1.98μs	7.15μs	0.604	0	0	41.72 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	678μs	3.54μs	21.8μs	0.316	0	0	41.75 KB
master	`WriteAndFlushEnrichedTraces`	net472	842μs	3.75μs	14μs	8.62	2.59	0.431	53.31 KB
#6536	`WriteAndFlushEnrichedTraces`	net6.0	592μs	3.39μs	25.4μs	0.604	0	0	41.86 KB
#6536	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	693μs	4μs	32.3μs	0.329	0	0	41.8 KB
#6536	`WriteAndFlushEnrichedTraces`	net472	849μs	3.42μs	13.3μs	8.08	2.55	0.425	53.28 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.4μs	1.78ns	6.89ns	0.014	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.85μs	1.1ns	4.11ns	0.0139	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.08μs	1.71ns	6.4ns	0.156	0.00103	987 B
#6536	`ExecuteNonQuery`	net6.0	1.28μs	3.88ns	15ns	0.0142	0	1.02 KB
#6536	`ExecuteNonQuery`	netcoreapp3.1	1.77μs	1.6ns	6.19ns	0.0133	0	1.02 KB
#6536	`ExecuteNonQuery`	net472	2.03μs	2.25ns	8.43ns	0.156	0.00101	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.24μs	0.817ns	3.17ns	0.0137	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.57μs	0.726ns	2.71ns	0.0134	976 B
master	`CallElasticsearch`	net472	2.55μs	1.53ns	5.71ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.29μs	0.754ns	2.92ns	0.0129	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.63μs	0.978ns	3.79ns	0.0139	1.02 KB
master	`CallElasticsearchAsync`	net472	2.57μs	1.83ns	7.08ns	0.167	1.05 KB
#6536	`CallElasticsearch`	net6.0	1.28μs	0.479ns	1.79ns	0.0134	976 B
#6536	`CallElasticsearch`	netcoreapp3.1	1.54μs	0.823ns	3.19ns	0.0132	976 B
#6536	`CallElasticsearch`	net472	2.44μs	2.53ns	9.81ns	0.157	995 B
#6536	`CallElasticsearchAsync`	net6.0	1.36μs	0.998ns	3.73ns	0.0129	952 B
#6536	`CallElasticsearchAsync`	netcoreapp3.1	1.59μs	0.935ns	3.5ns	0.0139	1.02 KB
#6536	`CallElasticsearchAsync`	net472	2.64μs	2.1ns	8.14ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.27μs	1.16ns	4.34ns	0.0134	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.53μs	1.64ns	5.91ns	0.0133	952 B
master	`ExecuteAsync`	net472	1.93μs	0.401ns	1.39ns	0.144	915 B
#6536	`ExecuteAsync`	net6.0	1.26μs	0.844ns	3.04ns	0.0132	952 B
#6536	`ExecuteAsync`	netcoreapp3.1	1.58μs	1.36ns	5.26ns	0.0127	952 B
#6536	`ExecuteAsync`	net472	1.81μs	0.405ns	1.46ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.38μs	2.19ns	8.5ns	0.031	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.45μs	1.83ns	6.85ns	0.0379	2.85 KB
master	`SendAsync`	net472	7.39μs	3.2ns	12.4ns	0.492	3.12 KB
#6536	`SendAsync`	net6.0	4.64μs	1.84ns	7.14ns	0.0321	2.31 KB
#6536	`SendAsync`	netcoreapp3.1	5.49μs	2.16ns	8.09ns	0.0358	2.85 KB
#6536	`SendAsync`	net472	7.36μs	1.35ns	5.04ns	0.493	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.52μs	0.687ns	2.66ns	0.023	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.21μs	0.989ns	3.57ns	0.0225	1.64 KB
master	`EnrichedLog`	net472	2.86μs	1.97ns	7.63ns	0.249	1.57 KB
#6536	`EnrichedLog`	net6.0	1.48μs	0.575ns	2.15ns	0.0232	1.64 KB
#6536	`EnrichedLog`	netcoreapp3.1	2.25μs	0.884ns	3.19ns	0.0214	1.64 KB
#6536	`EnrichedLog`	net472	2.7μs	3.51ns	13.6ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	118μs	258ns	932ns	0	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	121μs	232ns	900ns	0.0613	0	4.28 KB
master	`EnrichedLog`	net472	152μs	118ns	458ns	0.683	0.228	4.46 KB
#6536	`EnrichedLog`	net6.0	119μs	162ns	626ns	0.0596	0	4.28 KB
#6536	`EnrichedLog`	netcoreapp3.1	121μs	322ns	1.25μs	0	0	4.28 KB
#6536	`EnrichedLog`	net472	152μs	237ns	920ns	0.682	0.227	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.97μs	15.8ns	79ns	0.0304	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.18μs	1.68ns	6.28ns	0.0292	2.2 KB
master	`EnrichedLog`	net472	5.18μs	1.24ns	4.8ns	0.319	2.02 KB
#6536	`EnrichedLog`	net6.0	3μs	0.99ns	3.83ns	0.0299	2.2 KB
#6536	`EnrichedLog`	netcoreapp3.1	4.34μs	1.71ns	6.61ns	0.0282	2.2 KB
#6536	`EnrichedLog`	net472	4.9μs	1.11ns	4.3ns	0.32	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.35μs	0.594ns	2.3ns	0.0161	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.68μs	0.561ns	2.17ns	0.0149	1.14 KB
master	`SendReceive`	net472	2.14μs	0.793ns	3.07ns	0.183	1.16 KB
#6536	`SendReceive`	net6.0	1.37μs	1.23ns	4.62ns	0.0164	1.14 KB
#6536	`SendReceive`	netcoreapp3.1	1.76μs	0.77ns	2.98ns	0.0149	1.14 KB
#6536	`SendReceive`	net472	2.04μs	1.25ns	4.66ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.73μs	0.698ns	2.7ns	0.0219	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	4.1μs	1.84ns	7.13ns	0.0226	1.65 KB
master	`EnrichedLog`	net472	4.34μs	1.98ns	7.15ns	0.322	2.04 KB
#6536	`EnrichedLog`	net6.0	2.68μs	1ns	3.89ns	0.0215	1.6 KB
#6536	`EnrichedLog`	netcoreapp3.1	3.92μs	1.11ns	4.14ns	0.0214	1.65 KB
#6536	`EnrichedLog`	net472	4.37μs	3.95ns	15.3ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6536

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.164	569.09	488.71

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	452ns	0.246ns	0.951ns	0.00818	576 B
master	`StartFinishSpan`	netcoreapp3.1	615ns	1.03ns	3.98ns	0.00804	576 B
master	`StartFinishSpan`	net472	672ns	0.322ns	1.25ns	0.0916	578 B
master	`StartFinishScope`	net6.0	569ns	0.439ns	1.7ns	0.0097	696 B
master	`StartFinishScope`	netcoreapp3.1	720ns	0.908ns	3.52ns	0.00955	696 B
master	`StartFinishScope`	net472	943ns	0.769ns	2.88ns	0.104	658 B
#6536	`StartFinishSpan`	net6.0	467ns	0.306ns	1.19ns	0.00802	576 B
#6536	`StartFinishSpan`	netcoreapp3.1	602ns	0.981ns	3.8ns	0.00772	576 B
#6536	`StartFinishSpan`	net472	695ns	0.734ns	2.84ns	0.0917	578 B
#6536	`StartFinishScope`	net6.0	489ns	0.347ns	1.34ns	0.00972	696 B
#6536	`StartFinishScope`	netcoreapp3.1	726ns	0.32ns	1.24ns	0.00935	696 B
#6536	`StartFinishScope`	net472	911ns	0.492ns	1.9ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	659ns	0.43ns	1.67ns	0.00961	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	934ns	0.414ns	1.43ns	0.00906	696 B
master	`RunOnMethodBegin`	net472	1.04μs	0.671ns	2.42ns	0.104	658 B
#6536	`RunOnMethodBegin`	net6.0	653ns	0.461ns	1.78ns	0.00979	696 B
#6536	`RunOnMethodBegin`	netcoreapp3.1	896ns	0.451ns	1.75ns	0.00942	696 B
#6536	`RunOnMethodBegin`	net472	1.15μs	0.484ns	1.87ns	0.105	658 B

andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Jan 13, 2025

andrewlock requested a review from a team as a code owner January 13, 2025 13:24

andrewlock added 2 commits January 13, 2025 13:31

Add a Github action that ensures you correctly persist any changes to…

7128b3a

… the samples

Regenerate solution!

5400260

andrewlock force-pushed the andrew/ci/flag-missing-sample-build branch from b68df89 to 5400260 Compare January 13, 2025 14:10

kevingosse approved these changes Jan 13, 2025

View reviewed changes

lucaspimentel approved these changes Jan 13, 2025

View reviewed changes

andrewlock merged commit 23aa17e into master Jan 13, 2025
62 of 66 checks passed

andrewlock deleted the andrew/ci/flag-missing-sample-build branch January 13, 2025 17:19

github-actions bot added this to the vNext-v3 milestone Jan 13, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a Github action that ensures you correctly persist any changes to the samples #6536

Add a Github action that ensures you correctly persist any changes to the samples #6536

andrewlock commented Jan 13, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 13, 2025 •

edited

Loading

andrewlock commented Jan 13, 2025

andrewlock commented Jan 13, 2025

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6536

Raw results

Raw results

Add a Github action that ensures you correctly persist any changes to the samples #6536

Add a Github action that ensures you correctly persist any changes to the samples #6536

Conversation

andrewlock commented Jan 13, 2025 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

datadog-ddstaging bot commented Jan 13, 2025 • edited Loading

Datadog Report

andrewlock commented Jan 13, 2025

Execution-Time Benchmarks Report ⏱️

andrewlock commented Jan 13, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6536

Raw results

Raw results

andrewlock commented Jan 13, 2025 •

edited

Loading

datadog-ddstaging bot commented Jan 13, 2025 •

edited

Loading