Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding Delay to CheckForSmoke to Prevent Flakes #6396

Merged
merged 2 commits into from
Dec 4, 2024
Merged

Adding Delay to CheckForSmoke to Prevent Flakes #6396

merged 2 commits into from
Dec 4, 2024

Conversation

link04
Copy link
Contributor

@link04 link04 commented Dec 3, 2024

Summary of changes

There are a couple of NoException tests that make use of CheckForSmoke failing after app shutdown so adding delay to try testing

Reason for change

To try preventing some flake.

@link04 link04 requested a review from a team as a code owner December 3, 2024 22:26
@github-actions github-actions bot added the area:tests unit tests, integration tests label Dec 3, 2024
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Dec 3, 2024 •
edited
Loading

Datadog Report

Branch report: maximo/adding-delay-in-CheckForSmoke
Commit report: b1ab0ae
Test service: dd-trace-dotnet

✅ 0 Failed, 451469 Passed, 2736 Skipped, 19h 31m 13.1s Total Time

@andrewlock
Copy link
Member

andrewlock commented Dec 3, 2024 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6396) - mean (69ms)  : 66, 72
     .   : milestone, 69,
    master - mean (69ms)  : 66, 71
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6396) - mean (978ms)  : 953, 1003
     .   : milestone, 978,
    master - mean (977ms)  : 957, 998
     .   : milestone, 977,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6396) - mean (109ms)  : 107, 111
     .   : milestone, 109,
    master - mean (108ms)  : 106, 110
     .   : milestone, 108,

    section CallTarget+Inlining+NGEN
    This PR (6396) - mean (683ms)  : 665, 701
     .   : milestone, 683,
    master - mean (679ms)  : 665, 692
     .   : milestone, 679,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6396) - mean (92ms)  : 89, 94
     .   : milestone, 92,
    master - mean (91ms)  : 89, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6396) - mean (634ms)  : 617, 651
     .   : milestone, 634,
    master - mean (633ms)  : 617, 648
     .   : milestone, 633,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6396) - mean (190ms)  : 186, 195
     .   : milestone, 190,
    master - mean (190ms)  : 186, 194
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (6396) - mean (1,099ms)  : 1061, 1137
     .   : milestone, 1099,
    master - mean (1,092ms)  : 1055, 1128
     .   : milestone, 1092,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6396) - mean (277ms)  : 272, 281
     .   : milestone, 277,
    master - mean (277ms)  : 273, 281
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (6396) - mean (876ms)  : 852, 899
     .   : milestone, 876,
    master - mean (869ms)  : 842, 895
     .   : milestone, 869,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6396) - mean (266ms)  : 261, 271
     .   : milestone, 266,
    master - mean (265ms)  : 261, 269
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (6396) - mean (859ms)  : 826, 892
     .   : milestone, 859,
    master - mean (852ms)  : 818, 886
     .   : milestone, 852,
Loading

@andrewlock
Copy link
Member

andrewlock commented Dec 3, 2024 •
edited
Loading

Benchmarks Report for tracer 🐌

Benchmarks for #6396 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.124
  • 1 benchmarks are slower, with geometric mean 1.230
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.11μs 46.2ns 339ns 0.0204 0.00817 0 5.61 KB
master StartStopWithChild netcoreapp3.1 10μs 55.7ns 370ns 0.0242 0.0145 0.00483 5.8 KB
master StartStopWithChild net472 16.1μs 54.7ns 212ns 1.04 0.295 0.104 6.21 KB
#6396 StartStopWithChild net6.0 8μs 42ns 201ns 0.0198 0.00397 0 5.61 KB
#6396 StartStopWithChild netcoreapp3.1 10.3μs 53.2ns 266ns 0.0199 0.00996 0 5.8 KB
#6396 StartStopWithChild net472 16.2μs 56.5ns 219ns 1.04 0.308 0.0974 6.22 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 494μs 221ns 796ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 663μs 351ns 1.36μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 849μs 299ns 1.16μs 0.425 0 0 3.3 KB
#6396 WriteAndFlushEnrichedTraces net6.0 488μs 189ns 707ns 0 0 0 2.7 KB
#6396 WriteAndFlushEnrichedTraces netcoreapp3.1 647μs 402ns 1.5μs 0 0 0 2.7 KB
#6396 WriteAndFlushEnrichedTraces net472 854μs 636ns 2.46μs 0.428 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 147μs 845ns 7.36μs 0.142 0 0 14.47 KB
master SendRequest netcoreapp3.1 160μs 769ns 5.04μs 0.156 0 0 17.27 KB
master SendRequest net472 0.000368ns 0.000204ns 0.000763ns 0 0 0 0 b
#6396 SendRequest net6.0 150μs 854ns 5.91μs 0.152 0 0 14.47 KB
#6396 SendRequest netcoreapp3.1 164μs 920ns 6.38μs 0.158 0 0 17.27 KB
#6396 SendRequest net472 0.00288ns 0.00132ns 0.00512ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 561μs 2.23μs 8.02μs 0.571 0 0 41.72 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 720μs 4.04μs 25.5μs 0.351 0 0 41.7 KB
master WriteAndFlushEnrichedTraces net472 859μs 3.3μs 12.3μs 8.45 2.53 0.422 53.27 KB
#6396 WriteAndFlushEnrichedTraces net6.0 583μs 3.15μs 18.9μs 0.584 0 0 41.61 KB
#6396 WriteAndFlushEnrichedTraces netcoreapp3.1 669μs 2.85μs 13.7μs 0.324 0 0 41.73 KB
#6396 WriteAndFlushEnrichedTraces net472 903μs 2.96μs 11.5μs 8.36 2.64 0.44 53.3 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.33μs 0.902ns 3.49ns 0.014 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.8μs 2.42ns 9.36ns 0.0139 0 0 1.02 KB
master ExecuteNonQuery net472 2μs 4.01ns 15.5ns 0.156 0.00101 0 987 B
#6396 ExecuteNonQuery net6.0 1.35μs 1.59ns 6.17ns 0.0147 0 0 1.02 KB
#6396 ExecuteNonQuery netcoreapp3.1 1.76μs 0.88ns 3.29ns 0.0133 0 0 1.02 KB
#6396 ExecuteNonQuery net472 2.02μs 2.05ns 7.95ns 0.157 0.00102 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.21μs 0.54ns 2.02ns 0.0134 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.53μs 0.89ns 3.45ns 0.0129 0 0 976 B
master CallElasticsearch net472 2.62μs 1.55ns 5.6ns 0.157 0 0 995 B
master CallElasticsearchAsync net6.0 1.29μs 0.947ns 3.54ns 0.0134 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.7μs 1.49ns 5.78ns 0.0137 0 0 1.02 KB
master CallElasticsearchAsync net472 2.66μs 1.49ns 5.57ns 0.167 0 0 1.05 KB
#6396 CallElasticsearch net6.0 1.16μs 0.506ns 1.89ns 0.0135 0 0 976 B
#6396 CallElasticsearch netcoreapp3.1 1.6μs 5.46ns 21.1ns 0.0127 0 0 976 B
#6396 CallElasticsearch net472 2.57μs 1.53ns 5.72ns 0.158 0 0 995 B
#6396 CallElasticsearchAsync net6.0 1.28μs 0.859ns 3.21ns 0.0135 0 0 952 B
#6396 CallElasticsearchAsync netcoreapp3.1 1.61μs 0.793ns 3.07ns 0.0137 0 0 1.02 KB
#6396 CallElasticsearchAsync net472 2.53μs 1.59ns 6.15ns 0.166 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.44μs 0.812ns 3.04ns 0.013 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.63μs 1.83ns 6.84ns 0.013 0 0 952 B
master ExecuteAsync net472 1.91μs 0.65ns 2.43ns 0.145 0 0 915 B
#6396 ExecuteAsync net6.0 1.32μs 0.528ns 1.98ns 0.0133 0 0 952 B
#6396 ExecuteAsync netcoreapp3.1 1.62μs 1.74ns 6.52ns 0.0129 0 0 952 B
#6396 ExecuteAsync net472 1.81μs 0.509ns 1.97ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.42μs 1.75ns 6.77ns 0.0332 0 0 2.31 KB
master SendAsync netcoreapp3.1 5.28μs 1.7ns 6.57ns 0.0397 0 0 2.85 KB
master SendAsync net472 7.27μs 2.01ns 7.79ns 0.494 0 0 3.12 KB
#6396 SendAsync net6.0 4.34μs 2.12ns 8.19ns 0.0326 0 0 2.31 KB
#6396 SendAsync netcoreapp3.1 5.18μs 3.03ns 11.7ns 0.0362 0 0 2.85 KB
#6396 SendAsync net472 7.28μs 1.8ns 6.97ns 0.494 0 0 3.12 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.47μs 0.613ns 2.37ns 0.0235 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.21μs 5.23ns 20.3ns 0.0219 0 0 1.64 KB
master EnrichedLog net472 2.76μs 2.64ns 10.2ns 0.249 0 0 1.57 KB
#6396 EnrichedLog net6.0 1.45μs 1.03ns 3.98ns 0.0226 0 0 1.64 KB
#6396 EnrichedLog netcoreapp3.1 2.15μs 2.05ns 7.67ns 0.0225 0 0 1.64 KB
#6396 EnrichedLog net472 2.58μs 1.03ns 3.71ns 0.249 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 119μs 166ns 642ns 0.0598 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 124μs 99.7ns 359ns 0.0627 0 0 4.28 KB
master EnrichedLog net472 152μs 60.7ns 235ns 0.686 0.229 0 4.46 KB
#6396 EnrichedLog net6.0 120μs 227ns 880ns 0.0595 0 0 4.28 KB
#6396 EnrichedLog netcoreapp3.1 124μs 144ns 558ns 0 0 0 4.28 KB
#6396 EnrichedLog net472 151μs 225ns 870ns 0.678 0.226 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.04μs 1.37ns 5.32ns 0.0302 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.26μs 1.24ns 4.64ns 0.0297 0 0 2.2 KB
master EnrichedLog net472 4.93μs 1.71ns 6.61ns 0.32 0 0 2.02 KB
#6396 EnrichedLog net6.0 3.13μs 1.54ns 5.76ns 0.0298 0 0 2.2 KB
#6396 EnrichedLog netcoreapp3.1 4.33μs 0.963ns 3.73ns 0.0306 0 0 2.2 KB
#6396 EnrichedLog net472 4.84μs 1.25ns 4.82ns 0.319 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.25μs 0.349ns 1.35ns 0.0162 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.82μs 0.984ns 3.81ns 0.0154 0 0 1.14 KB
master SendReceive net472 2.07μs 0.763ns 2.86ns 0.183 0 0 1.16 KB
#6396 SendReceive net6.0 1.34μs 0.684ns 2.47ns 0.0164 0 0 1.14 KB
#6396 SendReceive netcoreapp3.1 1.78μs 1.09ns 3.79ns 0.0157 0 0 1.14 KB
#6396 SendReceive net472 2.14μs 1.32ns 5.12ns 0.183 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.69μs 1.31ns 5.06ns 0.0216 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.97μs 1.25ns 4.69ns 0.0216 0 0 1.65 KB
master EnrichedLog net472 4.39μs 2.49ns 9.64ns 0.322 0 0 2.04 KB
#6396 EnrichedLog net6.0 2.72μs 7.01ns 27.1ns 0.0219 0 0 1.6 KB
#6396 EnrichedLog netcoreapp3.1 3.88μs 1.72ns 6.67ns 0.0213 0 0 1.65 KB
#6396 EnrichedLog net472 4.38μs 21.2ns 84.7ns 0.324 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6396

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1 1.230 663.04 815.36

Faster 🎉 in #6396

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472 1.124 913.80 813.19

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 396ns 0.85ns 3.29ns 0.00808 0 0 576 B
master StartFinishSpan netcoreapp3.1 589ns 0.507ns 1.83ns 0.00793 0 0 576 B
master StartFinishSpan net472 728ns 1.46ns 5.66ns 0.0917 0 0 578 B
master StartFinishScope net6.0 490ns 0.582ns 2.25ns 0.00976 0 0 696 B
master StartFinishScope netcoreapp3.1 663ns 1.32ns 5.1ns 0.00915 0 0 696 B
master StartFinishScope net472 914ns 2.22ns 8.58ns 0.104 0 0 658 B
#6396 StartFinishSpan net6.0 402ns 0.578ns 2.24ns 0.00808 0 0 576 B
#6396 StartFinishSpan netcoreapp3.1 556ns 0.719ns 2.59ns 0.00791 0 0 576 B
#6396 StartFinishSpan net472 713ns 1.82ns 7.04ns 0.0915 0 0 578 B
#6396 StartFinishScope net6.0 500ns 0.641ns 2.48ns 0.0097 0 0 696 B
#6396 StartFinishScope netcoreapp3.1 815ns 1.4ns 5.43ns 0.00932 0 0 696 B
#6396 StartFinishScope net472 811ns 1.86ns 7.22ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 687ns 1.32ns 5.13ns 0.00957 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 903ns 1.55ns 5.98ns 0.00936 0 0 696 B
master RunOnMethodBegin net472 1.07μs 2ns 7.75ns 0.104 0 0 658 B
#6396 RunOnMethodBegin net6.0 624ns 1.33ns 5.15ns 0.00978 0 0 696 B
#6396 RunOnMethodBegin netcoreapp3.1 931ns 1.09ns 4.21ns 0.00943 0 0 696 B
#6396 RunOnMethodBegin net472 1.08μs 2.77ns 10.7ns 0.104 0 0 658 B

andrewlock
andrewlock approved these changes Dec 4, 2024
View reviewed changes
Copy link
Member

@andrewlock andrewlock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙈

@link04 link04 merged commit 71235e5 into master Dec 4, 2024
104 of 106 checks passed
@link04 link04 deleted the maximo/adding-delay-in-CheckForSmoke branch December 4, 2024 18:32
@github-actions github-actions bot added this to the vNext-v3 milestone Dec 4, 2024
@link04 link04 restored the maximo/adding-delay-in-CheckForSmoke branch December 5, 2024 14:39
veerbia pushed a commit that referenced this pull request Dec 16, 2024
@link04 @veerbia
Adding Delay to CheckForSmoke to Prevent Flakes (#6396)
5c95e5d 
## Summary of changes
There are a couple of `NoException` tests that make use of
`CheckForSmoke` failing after app shutdown so adding delay to try
testing

## Reason for change
To try preventing some flake.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewlock andrewlock andrewlock approved these changes

Assignees
No one assigned
Labels
area:tests unit tests, integration tests
Projects
None yet
Milestone
3.7.0
Development

Successfully merging this pull request may close these issues.
2 participants
@link04 @andrewlock