Disable tiered compilation in integration tests #3479

andrewlock · 2022-11-17T10:17:19Z

Summary of changes

Temporarily disable tiered compilation by setting COMPlus_TieredCompilation=0

Reason for change

We have been seeing flakiness in our integration tests for a long time which we have narrowed down to being related to how tiered JIT handles call-counting for our Rejited methods. This appears to be much more significant in our new Windows-2022 runners. We have raised an issue with the .NET runtime team. Disabling tiered compilation for now will hopefully significantly reduce flake in the meantime.

Implementation details

Set COMPlus_TieredCompilation=0 in integration sample apps. This is not a long-term solution, just a temporary fix for flake

Other details

For more details and analysis, see dotnet/runtime#77973

datadog-ddstaging · 2022-11-17T10:33:55Z

Datadog Report

Branch report: andrew/disable-tiered-jit
Commit report: 52cd15b

✅ dd-trace-dotnet 0 Failed, 0 New Flaky, 4 Passed, 0 Skipped, 32.39s Wall Time

datadog-ddstaging · 2022-11-17T10:33:56Z

Datadog Report

Branch report: andrew/disable-tiered-jit
Commit report: 59fc775

❄️ dd-trace-dotnet 0 Failed, 1 New Flaky, 235465 Passed, 1285 Skipped, 19m 41s Wall Time

New Flaky Tests (1)

TestIastNotWeakRequest - Datadog.Trace.Security.IntegrationTests.Iast.AspNetCore5IastTests - Last Failure

Expand for error

 Results do not match.
 Differences:
 Received: Iast.NotWeak.AspNetCore5.IastEnabled.received.txt
 Verified: Iast.NotWeak.AspNetCore5.IastEnabled.verified.txt
 Received Content:
 [
   {
     TraceId: Id_1,
     SpanId: Id_2,
     Name: aspnet_core.request,
 ...

andrewlock · 2022-11-17T11:46:52Z

Benchmarks Report 🐌

Benchmarks for #3479 compared to master:

All benchmarks have the same speed
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net472	776μs	1.09μs	4.09μs	0.383	3.22 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	556μs	298ns	1.15μs	0	2.63 KB
#3479	`WriteAndFlushEnrichedTraces`	net472	775μs	996ns	3.86μs	0.381	3.22 KB
#3479	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	555μs	261ns	1.01μs	0	2.63 KB

Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net472	182ns	0.0377ns	0.141ns	0.0675	0	425 B
master	`AllCycleSimpleBody`	netcoreapp3.1	244ns	0.569ns	2.2ns	0.00577	0	424 B
master	`AllCycleMoreComplexBody`	net472	180ns	0.0875ns	0.327ns	0.0637	0	401 B
master	`AllCycleMoreComplexBody`	netcoreapp3.1	241ns	0.345ns	1.34ns	0.00551	0	400 B
master	`BodyExtractorSimpleBody`	net472	283ns	0.326ns	1.26ns	0.0573	0	361 B
master	`BodyExtractorSimpleBody`	netcoreapp3.1	245ns	0.173ns	0.646ns	0.00369	0	272 B
master	`BodyExtractorMoreComplexBody`	net472	15.7μs	21.3ns	82.3ns	1.21	0.0156	7.62 KB
master	`BodyExtractorMoreComplexBody`	netcoreapp3.1	12.8μs	6.1ns	23.6ns	0.0894	0	6.75 KB
#3479	`AllCycleSimpleBody`	net472	182ns	0.0426ns	0.159ns	0.0676	0	425 B
#3479	`AllCycleSimpleBody`	netcoreapp3.1	245ns	0.155ns	0.599ns	0.0058	0	424 B
#3479	`AllCycleMoreComplexBody`	net472	181ns	0.036ns	0.13ns	0.0637	0	401 B
#3479	`AllCycleMoreComplexBody`	netcoreapp3.1	237ns	0.142ns	0.552ns	0.00549	0	400 B
#3479	`BodyExtractorSimpleBody`	net472	273ns	0.134ns	0.501ns	0.0574	0	361 B
#3479	`BodyExtractorSimpleBody`	netcoreapp3.1	250ns	0.121ns	0.454ns	0.00364	0	272 B
#3479	`BodyExtractorMoreComplexBody`	net472	15.3μs	13.7ns	51.4ns	1.2	0.0153	7.62 KB
#3479	`BodyExtractorMoreComplexBody`	netcoreapp3.1	12.9μs	4.76ns	17.8ns	0.0907	0	6.75 KB

Benchmarks.Trace.AspNetCoreBenchmark - Unknown 🤷 Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
master	`SendRequest`	netcoreapp3.1	176μs	234ns	906ns	0.263	20.44 KB
#3479	`SendRequest`	net472	0ns	0ns	0ns	0	0 b
#3479	`SendRequest`	netcoreapp3.1	174μs	194ns	750ns	0.26	20.44 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net472	1.77μs	1.14ns	4.43ns	0.16	0.000891	1.01 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.42μs	0.782ns	2.82ns	0.0135	0	1 KB
#3479	`ExecuteNonQuery`	net472	1.74μs	0.777ns	3.01ns	0.16	0.000867	1.01 KB
#3479	`ExecuteNonQuery`	netcoreapp3.1	1.4μs	0.569ns	2.2ns	0.0133	0	1 KB

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net472	2.51μs	1.27ns	4.92ns	0.193	1.22 KB
master	`CallElasticsearch`	netcoreapp3.1	1.5μs	0.373ns	1.35ns	0.0151	1.16 KB
master	`CallElasticsearchAsync`	net472	2.67μs	1.67ns	6.47ns	0.214	1.36 KB
master	`CallElasticsearchAsync`	netcoreapp3.1	1.64μs	0.55ns	2.13ns	0.0173	1.28 KB
#3479	`CallElasticsearch`	net472	2.49μs	0.6ns	2.25ns	0.193	1.22 KB
#3479	`CallElasticsearch`	netcoreapp3.1	1.51μs	0.686ns	2.66ns	0.0159	1.16 KB
#3479	`CallElasticsearchAsync`	net472	2.62μs	1.29ns	4.84ns	0.215	1.36 KB
#3479	`CallElasticsearchAsync`	netcoreapp3.1	1.63μs	0.53ns	1.91ns	0.017	1.28 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net472	2.69μs	0.855ns	3.31ns	0.236	1.49 KB
master	`ExecuteAsync`	netcoreapp3.1	1.73μs	1.64ns	6.35ns	0.0192	1.41 KB
#3479	`ExecuteAsync`	net472	2.76μs	1.58ns	6.11ns	0.235	1.49 KB
#3479	`ExecuteAsync`	netcoreapp3.1	1.76μs	0.581ns	2.25ns	0.0184	1.41 KB

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net472	5.88μs	1.29ns	4.64ns	0.448	2.83 KB
master	`SendAsync`	netcoreapp3.1	3.67μs	1.33ns	5.15ns	0.0367	2.66 KB
#3479	`SendAsync`	net472	5.85μs	2.06ns	7.99ns	0.449	2.83 KB
#3479	`SendAsync`	netcoreapp3.1	3.56μs	3.17ns	11.9ns	0.0361	2.66 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net472	2.89μs	2.7ns	10.1ns	0.298	1.88 KB
master	`EnrichedLog`	netcoreapp3.1	2.39μs	1.12ns	4.36ns	0.0252	1.91 KB
#3479	`EnrichedLog`	net472	2.89μs	2.61ns	10.1ns	0.297	1.88 KB
#3479	`EnrichedLog`	netcoreapp3.1	2.38μs	0.802ns	3.11ns	0.026	1.91 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net472	150μs	91.2ns	341ns	0.675	0.225	4.72 KB
master	`EnrichedLog`	netcoreapp3.1	120μs	223ns	865ns	0	0	4.55 KB
#3479	`EnrichedLog`	net472	149μs	93.2ns	361ns	0.745	0.223	4.72 KB
#3479	`EnrichedLog`	netcoreapp3.1	118μs	126ns	489ns	0.059	0	4.55 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net472	5.56μs	1.65ns	6.4ns	0.579	0.00278	3.65 KB
master	`EnrichedLog`	netcoreapp3.1	4.24μs	2.27ns	8.49ns	0.0529	0	3.98 KB
#3479	`EnrichedLog`	net472	5.49μs	1.87ns	6.99ns	0.58	0.00275	3.65 KB
#3479	`EnrichedLog`	netcoreapp3.1	4.26μs	1.08ns	4.06ns	0.0534	0	3.98 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net472	2.28μs	0.772ns	2.89ns	0.228	1.44 KB
master	`SendReceive`	netcoreapp3.1	1.84μs	0.773ns	2.89ns	0.0192	1.38 KB
#3479	`SendReceive`	net472	2.29μs	2.56ns	9.59ns	0.228	1.44 KB
#3479	`SendReceive`	netcoreapp3.1	1.81μs	0.815ns	3.16ns	0.0191	1.38 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net472	4.8μs	1.54ns	5.95ns	0.364	2.3 KB
master	`EnrichedLog`	netcoreapp3.1	3.95μs	0.832ns	3ns	0.0256	1.86 KB
#3479	`EnrichedLog`	net472	4.78μs	1.5ns	5.81ns	0.362	2.3 KB
#3479	`EnrichedLog`	netcoreapp3.1	4.21μs	1.08ns	4.05ns	0.0249	1.86 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net472	1.15μs	0.386ns	1.49ns	0.139	875 B
master	`StartFinishSpan`	netcoreapp3.1	913ns	0.513ns	1.99ns	0.011	824 B
master	`StartFinishScope`	net472	1.32μs	0.295ns	1.06ns	0.152	955 B
master	`StartFinishScope`	netcoreapp3.1	1.07μs	0.308ns	1.19ns	0.0128	944 B
#3479	`StartFinishSpan`	net472	1.19μs	1.02ns	3.93ns	0.139	875 B
#3479	`StartFinishSpan`	netcoreapp3.1	969ns	0.686ns	2.66ns	0.0111	824 B
#3479	`StartFinishScope`	net472	1.35μs	0.559ns	2.09ns	0.152	955 B
#3479	`StartFinishScope`	netcoreapp3.1	1.08μs	0.487ns	1.89ns	0.0129	944 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net472	1.4μs	0.48ns	1.86ns	0.152	955 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.14μs	0.327ns	1.13ns	0.0131	944 B
#3479	`RunOnMethodBegin`	net472	1.45μs	0.389ns	1.51ns	0.151	955 B
#3479	`RunOnMethodBegin`	netcoreapp3.1	1.17μs	0.488ns	1.83ns	0.0129	944 B

andrewlock · 2022-11-17T14:21:12Z

Code Coverage Report 📊

✔️ Merging #3479 into master will not change line coverage
✔️ Merging #3479 into master will not change branch coverage
⛔ Merging #3479 into master will will increase complexity by 78

	master	#3479	Change
Lines	`20547` / `27924`	`20523` / `27967`
Lines %	`74%`	`73%`	`0%` ✔️
Branches	`11890` / `17447`	`11925` / `17515`
Branches %	`68%`	`68%`	`0%` ✔️
Complexity	`18874`	`18952`	`78` ⛔

View the full report for further details:

Datadog.Trace Breakdown ✔️

	master	#3479	Change
Lines %	`74%`	`73%`	`0%` ✔️
Branches %	`68%`	`68%`	`0%` ✔️
Complexity	`18874`	`18952`	`78` ⛔

The following classes have significant coverage changes.

File	Line coverage change	Branch coverage change	Complexity change
Datadog.Trace.Ci.GitInfo	`-21%` ⛔	`-18%` ⛔	`0` ✔️
Datadog.Trace.Propagators.SpanContextPropagator	`-8%` ⛔	`-3%` ⚠️	`0` ✔️
Datadog.Trace.Propagators.DistributedContextExtractor	`-6%` ⛔	`-17%` ⛔	`1` ⛔
Datadog.Trace.Debugger.PInvoke.DebuggerNativeMethods	`-6%` ⛔	`-10%` ⛔	`0` ✔️

The following classes were added in #3479:

File	Line coverage	Branch coverage	Complexity
Datadog.Trace.Iast.OverheadController	`82%`	`100%`	`11`
Datadog.Trace.Propagators.B3MultipleHeaderContextPropagator	`100%`	`94%`	`33`
Datadog.Trace.Propagators.SpanContextPropagatorFactory	`100%`	`100%`	`21`
Datadog.Trace.Propagators.W3CTraceContextPropagator	`95%`	`93%`	`41`

3 classes were removed from Datadog.Trace in #3479

View the full reports for further details:

Disable tiered compilation everywhere

52cd15b

andrewlock added the area:tests unit tests, integration tests label Nov 17, 2022

andrewlock requested a review from a team as a code owner November 17, 2022 10:17

pierotibou approved these changes Nov 17, 2022

View reviewed changes

tonyredondo approved these changes Nov 17, 2022

View reviewed changes

This comment has been minimized.

Sign in to view

Add COMPlus_TieredCompilation=0 to smoke tests

59fc775

andrewlock requested review from a team as code owners November 17, 2022 11:18

andrewlock merged commit 22990ad into master Nov 17, 2022

andrewlock deleted the andrew/disable-tiered-jit branch November 17, 2022 14:06

github-actions bot added this to the vNext milestone Nov 17, 2022

andrewlock mentioned this pull request Nov 17, 2022

Add support for .NET 7 #3482

Merged

tonyredondo mentioned this pull request Nov 24, 2022

Adds a workaround for the dotnet runtime issue 77973 #3506

Merged

gleocadie mentioned this pull request Nov 29, 2022

[Profiler] Disable tiered compilation in integration tests #3519

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable tiered compilation in integration tests #3479

Disable tiered compilation in integration tests #3479

andrewlock commented Nov 17, 2022

datadog-ddstaging bot commented Nov 17, 2022

datadog-ddstaging bot commented Nov 17, 2022 •

edited

Loading

This comment has been minimized.

andrewlock commented Nov 17, 2022

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

andrewlock commented Nov 17, 2022

Disable tiered compilation in integration tests #3479

Disable tiered compilation in integration tests #3479

Conversation

andrewlock commented Nov 17, 2022

Summary of changes

Reason for change

Implementation details

Other details

datadog-ddstaging bot commented Nov 17, 2022

Datadog Report

datadog-ddstaging bot commented Nov 17, 2022 • edited Loading

Datadog Report

New Flaky Tests (1)

This comment has been minimized.

andrewlock commented Nov 17, 2022

Benchmarks Report 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

andrewlock commented Nov 17, 2022

Code Coverage Report 📊

Datadog.Trace Breakdown ✔️

datadog-ddstaging bot commented Nov 17, 2022 •

edited

Loading