Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stabilize criterion benchmark results #576

Merged
merged 1 commit into from
Aug 7, 2024
Merged

Stabilize criterion benchmark results #576

merged 1 commit into from
Aug 7, 2024

Conversation

bantonsson
Copy link
Contributor

@bantonsson bantonsson commented Aug 6, 2024 •
edited
Loading

What does this PR do?

This PR tries to stabilize the criterion micro benchmarks results. The reason why the benchmark result comment is so full of changes is that the batching and how the benchmarks are run has changed compared to main.

Motivation

There is soo much noise in the results that they trigger false positives on almost every PR.

Additional Notes

Anything else we should know when reviewing?

How to test the change?

I have run the the benchmarks repeatedly on a separate PR #577 (that has no code changes), and only occasionally will there be a change in a benchmark result.

@codecov-commenter
Copy link

codecov-commenter commented Aug 6, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.30%. Comparing base (66b83a6) to head (11db3f1).

Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #576      +/-   ##
==========================================
+ Coverage   71.26%   71.30%   +0.03%     
==========================================
  Files         220      220              
  Lines       29921    29904      -17     
==========================================
- Hits        21323    21322       -1     
+ Misses       8598     8582      -16
Components Coverage Δ
crashtracker 21.11% <ø> (-0.06%) ⬇️
datadog-alloc 98.73% <ø> (ø)
data-pipeline 50.00% <ø> (ø)
data-pipeline-ffi 0.00% <ø> (ø)
ddcommon 83.07% <ø> (ø)
ddcommon-ffi 70.20% <ø> (ø)
ddtelemetry 58.95% <ø> (ø)
ipc 84.18% <ø> (ø)
profiling 84.26% <ø> (ø)
profiling-ffi 77.42% <ø> (ø)
serverless 0.00% <ø> (ø)
sidecar 34.55% <ø> (ø)
sidecar-ffi 0.00% <ø> (ø)
spawn-worker 54.87% <ø> (ø)
trace-mini-agent 70.88% <ø> (ø)
trace-normalization 98.24% <ø> (ø)
trace-obfuscation 95.73% <ø> (ø)
trace-protobuf 77.16% <ø> (ø)
trace-utils 93.39% <ø> (+0.42%) ⬆️

@pr-commenter
Copy link

pr-commenter bot commented Aug 6, 2024 •
edited
Loading

Benchmarks

Comparison

Benchmark execution time: 2024-08-07 15:03:58

Comparing candidate commit 11db3f1 in PR branch ban/stabilize-criterion-results with baseline commit 66b83a6 in branch main.

Found 5 performance improvements and 17 performance regressions! Performance is the same for 1 metrics, 21 unstable metrics.

scenario:benching deserializing traces from msgpack to their internal representation

  • 🟩 execution_time [-105.668ns; -88.465ns] or [-7.275%; -6.090%]

scenario:credit_card/is_card_number/

  • 🟥 execution_time [+2.412µs; +2.413µs] or [+67252.259%; +67258.263%]
  • 🟩 throughput [+134810277.727op/s; +135390225.275op/s] or [+48.354%; +48.562%]

scenario:credit_card/is_card_number/ 3782-8224-6310-005

  • 🟥 throughput [-2360705.915op/s; -2287918.714op/s] or [-20.599%; -19.963%]

scenario:credit_card/is_card_number/ 378282246310005

  • 🟥 throughput [-2652497.989op/s; -2586343.462op/s] or [-21.841%; -21.296%]

scenario:credit_card/is_card_number/37828224631

  • 🟩 throughput [+135560968.105op/s; +135965721.118op/s] or [+48.751%; +48.897%]

scenario:credit_card/is_card_number/378282246310005

  • 🟥 throughput [-2790013.345op/s; -2703493.516op/s] or [-22.313%; -21.621%]

scenario:credit_card/is_card_number/37828224631000521389798

  • 🟥 throughput [-5527526.858op/s; -5490285.602op/s] or [-37.575%; -37.322%]

scenario:credit_card/is_card_number/x371413321323331

  • 🟥 throughput [-2519329.185op/s; -2172881.156op/s] or [-5.657%; -4.879%]

scenario:credit_card/is_card_number_no_luhn/

  • 🟥 execution_time [+2.413µs; +2.413µs] or [+70348.780%; +70356.013%]
  • 🟩 throughput [+122080815.610op/s; +122488916.858op/s] or [+41.866%; +42.006%]

scenario:credit_card/is_card_number_no_luhn/ 3782-8224-6310-005

  • 🟥 throughput [-3355537.922op/s; -3327349.865op/s] or [-23.188%; -22.993%]

scenario:credit_card/is_card_number_no_luhn/ 378282246310005

  • 🟥 throughput [-3915128.795op/s; -3851072.259op/s] or [-25.157%; -24.746%]

scenario:credit_card/is_card_number_no_luhn/37828224631

  • 🟥 execution_time [+2.413µs; +2.413µs] or [+70345.500%; +70351.592%]
  • 🟩 throughput [+122093445.460op/s; +122499169.095op/s] or [+41.875%; +42.014%]

scenario:credit_card/is_card_number_no_luhn/378282246310005

  • 🟥 throughput [-4404578.183op/s; -4354872.999op/s] or [-27.233%; -26.926%]

scenario:credit_card/is_card_number_no_luhn/37828224631000521389798

  • 🟥 throughput [-5560423.738op/s; -5494623.113op/s] or [-37.721%; -37.275%]

scenario:credit_card/is_card_number_no_luhn/x371413321323331

  • 🟥 throughput [-2909557.546op/s; -2578053.041op/s] or [-6.458%; -5.722%]

scenario:normalization/normalize_trace/test_trace

  • 🟥 execution_time [+15.069ns; +19.703ns] or [+4.568%; +5.973%]

scenario:redis/obfuscate_redis_string

  • 🟥 execution_time [+1.313µs; +1.565µs] or [+3.577%; +4.264%]

scenario:sql/obfuscate_sql_string

  • 🟥 execution_time [+6.407µs; +6.453µs] or [+9.434%; +9.501%]

scenario:tags/replace_trace_tags

  • 🟥 execution_time [+405.396ns; +409.991ns] or [+17.276%; +17.472%]

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 11db3f1 1723042292 ban/stabilize-criterion-results
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_trace/test_trace execution_time 332.750ns 347.265ns ± 15.516ns 340.943ns ± 5.543ns 351.680ns 381.017ns 393.452ns 393.641ns 15.46% 1.509 1.208 4.46% 1.097ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_trace/test_trace execution_time [345.115ns; 349.415ns] or [-0.619%; +0.619%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 11db3f1 1723042292 ban/stabilize-criterion-results
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching string interning on wordpress profile execution_time 134.949µs 135.401µs ± 0.197µs 135.386µs ± 0.110µs 135.503µs 135.667µs 135.908µs 136.641µs 0.93% 1.794 8.973 0.14% 0.014µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching string interning on wordpress profile execution_time [135.373µs; 135.428µs] or [-0.020%; +0.020%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 11db3f1 1723042292 ban/stabilize-criterion-results
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time 283.561µs 285.809µs ± 0.977µs 285.816µs ± 0.356µs 286.127µs 286.573µs 286.910µs 297.029µs 3.92% 7.459 85.458 0.34% 0.069µs 1 200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput 3366673.420op/s 3498877.647op/s ± 11665.240op/s 3498757.286op/s ± 4362.681op/s 3503787.705op/s 3511606.057op/s 3520226.856op/s 3526574.946op/s 0.80% -7.137 80.837 0.33% 824.857op/s 1 200
normalization/normalize_name/normalize_name/bad-name execution_time 36.902µs 37.100µs ± 0.077µs 37.094µs ± 0.048µs 37.153µs 37.231µs 37.288µs 37.346µs 0.68% 0.264 0.238 0.21% 0.005µs 1 200
normalization/normalize_name/normalize_name/bad-name throughput 26776953.893op/s 26954231.130op/s ± 55630.017op/s 26958208.817op/s ± 35274.104op/s 26991534.886op/s 27036400.928op/s 27076232.516op/s 27098786.773op/s 0.52% -0.250 0.230 0.21% 3933.636op/s 1 200
normalization/normalize_name/normalize_name/good execution_time 24.129µs 24.406µs ± 0.171µs 24.366µs ± 0.100µs 24.489µs 24.737µs 24.877µs 24.967µs 2.47% 0.879 0.262 0.70% 0.012µs 1 200
normalization/normalize_name/normalize_name/good throughput 40052734.853op/s 40974960.348op/s ± 285145.244op/s 41040527.014op/s ± 166994.674op/s 41168077.017op/s 41336273.484op/s 41386528.191op/s 41443905.674op/s 0.98% -0.848 0.190 0.69% 20162.814op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time [285.674µs; 285.945µs] or [-0.047%; +0.047%] None None None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput [3497260.957op/s; 3500494.337op/s] or [-0.046%; +0.046%] None None None
normalization/normalize_name/normalize_name/bad-name execution_time [37.089µs; 37.111µs] or [-0.029%; +0.029%] None None None
normalization/normalize_name/normalize_name/bad-name throughput [26946521.345op/s; 26961940.916op/s] or [-0.029%; +0.029%] None None None
normalization/normalize_name/normalize_name/good execution_time [24.383µs; 24.430µs] or [-0.097%; +0.097%] None None None
normalization/normalize_name/normalize_name/good throughput [40935441.960op/s; 41014478.736op/s] or [-0.096%; +0.096%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 11db3f1 1723042292 ban/stabilize-criterion-results
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
credit_card/is_card_number/ execution_time 2.415µs 2.416µs ± 0.001µs 2.416µs ± 0.001µs 2.417µs 2.417µs 2.419µs 2.420µs 0.17% 1.182 2.767 0.03% 0.000µs 1 200
credit_card/is_card_number/ throughput 413240224.791op/s 413896460.721op/s ± 132961.155op/s 413926765.072op/s ± 88007.037op/s 413990570.220op/s 414066041.572op/s 414092849.780op/s 414153728.700op/s 0.05% -1.179 2.751 0.03% 9401.773op/s 1 200
credit_card/is_card_number/ 3782-8224-6310-005 execution_time 105.272µs 109.529µs ± 2.864µs 109.710µs ± 2.773µs 112.238µs 113.320µs 113.709µs 115.390µs 5.18% -0.025 -1.544 2.61% 0.203µs 1 200
credit_card/is_card_number/ 3782-8224-6310-005 throughput 8666275.162op/s 9136250.919op/s ± 239198.414op/s 9114955.703op/s ± 225095.195op/s 9380148.057op/s 9467860.879op/s 9490949.015op/s 9499236.026op/s 4.22% 0.060 -1.557 2.61% 16913.882op/s 1 200
credit_card/is_card_number/ 378282246310005 execution_time 100.815µs 105.037µs ± 2.349µs 104.776µs ± 1.956µs 107.006µs 108.795µs 110.172µs 110.370µs 5.34% 0.267 -1.021 2.23% 0.166µs 1 200
credit_card/is_card_number/ 378282246310005 throughput 9060414.276op/s 9525208.042op/s ± 211860.256op/s 9544150.281op/s ± 180105.371op/s 9715033.701op/s 9827432.545op/s 9885944.244op/s 9919161.930op/s 3.93% -0.207 -1.061 2.22% 14980.782op/s 1 200
credit_card/is_card_number/37828224631 execution_time 2.415µs 2.416µs ± 0.005µs 2.416µs ± 0.001µs 2.417µs 2.417µs 2.419µs 2.485µs 2.87% 13.509 185.197 0.21% 0.000µs 1 200
credit_card/is_card_number/37828224631 throughput 402336291.706op/s 413830964.464op/s ± 827922.961op/s 413886507.300op/s ± 98149.706op/s 413997455.003op/s 414063956.908op/s 414144937.741op/s 414146707.193op/s 0.06% -13.479 184.654 0.20% 58542.994op/s 1 200
credit_card/is_card_number/378282246310005 execution_time 97.896µs 102.574µs ± 2.936µs 102.933µs ± 2.731µs 105.369µs 106.581µs 107.089µs 107.815µs 4.74% -0.061 -1.504 2.85% 0.208µs 1 200
credit_card/is_card_number/378282246310005 throughput 9275138.465op/s 9757033.487op/s ± 279900.635op/s 9715063.175op/s ± 257821.049op/s 10024931.919op/s 10175595.214op/s 10197200.896op/s 10214871.915op/s 5.14% 0.103 -1.501 2.86% 19791.964op/s 1 200
credit_card/is_card_number/37828224631000521389798 execution_time 105.820µs 108.684µs ± 0.995µs 108.590µs ± 0.631µs 109.245µs 110.525µs 110.947µs 111.075µs 2.29% 0.184 -0.114 0.91% 0.070µs 1 200
credit_card/is_card_number/37828224631000521389798 throughput 9002935.642op/s 9201782.767op/s ± 84141.701op/s 9208935.603op/s ± 53235.003op/s 9258882.566op/s 9343109.816op/s 9387491.900op/s 9449983.242op/s 2.62% -0.133 -0.105 0.91% 5949.717op/s 1 200
credit_card/is_card_number/x371413321323331 execution_time 22.584µs 23.715µs ± 0.498µs 23.672µs ± 0.345µs 24.054µs 24.518µs 24.839µs 24.983µs 5.54% 0.046 -0.325 2.09% 0.035µs 1 200
credit_card/is_card_number/x371413321323331 throughput 40027367.283op/s 42186183.829op/s ± 885727.361op/s 42244884.969op/s ± 617128.915op/s 42753381.892op/s 43795779.135op/s 44151598.132op/s 44278995.626op/s 4.82% 0.058 -0.330 2.09% 62630.382op/s 1 200
credit_card/is_card_number_no_luhn/ execution_time 2.415µs 2.416µs ± 0.001µs 2.416µs ± 0.000µs 2.417µs 2.417µs 2.418µs 2.423µs 0.30% 3.297 21.455 0.04% 0.000µs 1 200
credit_card/is_card_number_no_luhn/ throughput 412679541.524op/s 413880616.443op/s ± 153092.192op/s 413901138.228op/s ± 84125.675op/s 413979277.032op/s 414055624.879op/s 414086448.222op/s 414134027.139op/s 0.06% -3.284 21.319 0.04% 10825.253op/s 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time 88.843µs 89.851µs ± 0.374µs 89.821µs ± 0.122µs 89.935µs 90.545µs 91.085µs 91.565µs 1.94% 1.212 4.245 0.42% 0.026µs 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput 10921220.556op/s 11129733.200op/s ± 46150.787op/s 11133264.437op/s ± 15102.055op/s 11148201.563op/s 11197280.692op/s 11242048.366op/s 11255848.142op/s 1.10% -1.153 4.064 0.41% 3263.353op/s 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time 82.869µs 85.648µs ± 1.529µs 85.662µs ± 1.207µs 86.638µs 88.848µs 88.906µs 89.013µs 3.91% 0.414 -0.623 1.78% 0.108µs 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 throughput 11234329.191op/s 11679434.385op/s ± 207114.751op/s 11673727.110op/s ± 165568.178op/s 11855442.043op/s 11966585.316op/s 12031218.508op/s 12067238.654op/s 3.37% -0.350 -0.689 1.77% 14645.224op/s 1 200
credit_card/is_card_number_no_luhn/37828224631 execution_time 2.415µs 2.416µs ± 0.001µs 2.416µs ± 0.000µs 2.417µs 2.418µs 2.419µs 2.419µs 0.12% 0.819 0.758 0.03% 0.000µs 1 200
credit_card/is_card_number_no_luhn/37828224631 throughput 413407304.402op/s 413865256.519op/s ± 129054.452op/s 413886129.704op/s ± 85485.296op/s 413965161.750op/s 414037438.050op/s 414068112.831op/s 414124673.744op/s 0.06% -0.817 0.752 0.03% 9125.528op/s 1 200
credit_card/is_card_number_no_luhn/378282246310005 execution_time 80.975µs 84.799µs ± 0.879µs 85.045µs ± 0.493µs 85.406µs 85.819µs 86.147µs 86.338µs 1.52% -1.146 1.650 1.03% 0.062µs 1 200
credit_card/is_card_number_no_luhn/378282246310005 throughput 11582358.452op/s 11793910.596op/s ± 123739.585op/s 11758524.399op/s ± 67817.181op/s 11858221.121op/s 12020218.802op/s 12147905.181op/s 12349426.504op/s 5.03% 1.221 1.958 1.05% 8749.710op/s 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time 106.361µs 108.545µs ± 0.938µs 108.587µs ± 0.519µs 109.058µs 109.947µs 111.308µs 112.765µs 3.85% 0.478 1.895 0.86% 0.066µs 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput 8868025.191op/s 9213419.432op/s ± 79341.463op/s 9209196.405op/s ± 44025.452op/s 9261940.606op/s 9340625.305op/s 9399462.558op/s 9401970.006op/s 2.09% -0.385 1.638 0.86% 5610.289op/s 1 200
credit_card/is_card_number_no_luhn/x371413321323331 execution_time 22.361µs 23.646µs ± 0.523µs 23.631µs ± 0.384µs 24.019µs 24.483µs 24.710µs 24.773µs 4.83% -0.089 -0.577 2.21% 0.037µs 1 200
credit_card/is_card_number_no_luhn/x371413321323331 throughput 40366390.051op/s 42311723.153op/s ± 938312.509op/s 42317617.442op/s ± 676451.855op/s 42943187.981op/s 43943498.111op/s 44346631.897op/s 44721359.100op/s 5.68% 0.183 -0.545 2.21% 66348.714op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
credit_card/is_card_number/ execution_time [2.416µs; 2.416µs] or [-0.004%; +0.004%] None None None
credit_card/is_card_number/ throughput [413878033.583op/s; 413914887.858op/s] or [-0.004%; +0.004%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 execution_time [109.132µs; 109.926µs] or [-0.362%; +0.362%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 throughput [9103100.319op/s; 9169401.518op/s] or [-0.363%; +0.363%] None None None
credit_card/is_card_number/ 378282246310005 execution_time [104.711µs; 105.362µs] or [-0.310%; +0.310%] None None None
credit_card/is_card_number/ 378282246310005 throughput [9495846.248op/s; 9554569.836op/s] or [-0.308%; +0.308%] None None None
credit_card/is_card_number/37828224631 execution_time [2.416µs; 2.417µs] or [-0.028%; +0.028%] None None None
credit_card/is_card_number/37828224631 throughput [413716222.304op/s; 413945706.623op/s] or [-0.028%; +0.028%] None None None
credit_card/is_card_number/378282246310005 execution_time [102.167µs; 102.981µs] or [-0.397%; +0.397%] None None None
credit_card/is_card_number/378282246310005 throughput [9718241.951op/s; 9795825.023op/s] or [-0.398%; +0.398%] None None None
credit_card/is_card_number/37828224631000521389798 execution_time [108.546µs; 108.822µs] or [-0.127%; +0.127%] None None None
credit_card/is_card_number/37828224631000521389798 throughput [9190121.536op/s; 9213443.997op/s] or [-0.127%; +0.127%] None None None
credit_card/is_card_number/x371413321323331 execution_time [23.646µs; 23.784µs] or [-0.291%; +0.291%] None None None
credit_card/is_card_number/x371413321323331 throughput [42063430.535op/s; 42308937.122op/s] or [-0.291%; +0.291%] None None None
credit_card/is_card_number_no_luhn/ execution_time [2.416µs; 2.416µs] or [-0.005%; +0.005%] None None None
credit_card/is_card_number_no_luhn/ throughput [413859399.338op/s; 413901833.549op/s] or [-0.005%; +0.005%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time [89.799µs; 89.903µs] or [-0.058%; +0.058%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput [11123337.145op/s; 11136129.255op/s] or [-0.057%; +0.057%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time [85.436µs; 85.859µs] or [-0.247%; +0.247%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 throughput [11650730.272op/s; 11708138.498op/s] or [-0.246%; +0.246%] None None None
credit_card/is_card_number_no_luhn/37828224631 execution_time [2.416µs; 2.416µs] or [-0.004%; +0.004%] None None None
credit_card/is_card_number_no_luhn/37828224631 throughput [413847370.813op/s; 413883142.225op/s] or [-0.004%; +0.004%] None None None
credit_card/is_card_number_no_luhn/378282246310005 execution_time [84.677µs; 84.920µs] or [-0.144%; +0.144%] None None None
credit_card/is_card_number_no_luhn/378282246310005 throughput [11776761.479op/s; 11811059.712op/s] or [-0.145%; +0.145%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time [108.415µs; 108.675µs] or [-0.120%; +0.120%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput [9202423.468op/s; 9224415.395op/s] or [-0.119%; +0.119%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 execution_time [23.573µs; 23.718µs] or [-0.306%; +0.306%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 throughput [42181682.064op/s; 42441764.243op/s] or [-0.307%; +0.307%] None None None

Group 5

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 11db3f1 1723042292 ban/stabilize-criterion-results
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
redis/obfuscate_redis_string execution_time 37.606µs 38.140µs ± 0.909µs 37.735µs ± 0.041µs 37.774µs 40.080µs 40.133µs 41.049µs 8.78% 1.718 1.060 2.38% 0.064µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
redis/obfuscate_redis_string execution_time [38.014µs; 38.266µs] or [-0.330%; +0.330%] None None None

Group 6

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 11db3f1 1723042292 ban/stabilize-criterion-results
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 1.417µs 2.914µs ± 1.402µs 2.754µs ± 0.022µs 2.771µs 2.993µs 12.838µs 15.738µs 471.42% 7.923 62.735 47.98% 0.099µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [2.720µs; 3.108µs] or [-6.667%; +6.667%] None None None

Group 7

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 11db3f1 1723042292 ban/stabilize-criterion-results
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 17.095µs 23.206µs ± 16.207µs 17.305µs ± 0.087µs 17.535µs 52.552µs 57.587µs 152.295µs 780.05% 4.622 28.834 69.67% 1.146µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [20.960µs; 25.452µs] or [-9.679%; +9.679%] None None None

Group 8

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 11db3f1 1723042292 ban/stabilize-criterion-results
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching deserializing traces from msgpack to their internal representation execution_time 1.154µs 1.355µs ± 0.062µs 1.359µs ± 0.051µs 1.402µs 1.446µs 1.458µs 1.458µs 7.32% -0.608 0.643 4.57% 0.004µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching deserializing traces from msgpack to their internal representation execution_time [1.347µs; 1.364µs] or [-0.634%; +0.634%] None None None

Group 9

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 11db3f1 1723042292 ban/stabilize-criterion-results
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sql/obfuscate_sql_string execution_time 74.137µs 74.346µs ± 0.119µs 74.334µs ± 0.028µs 74.363µs 74.405µs 74.681µs 75.732µs 1.88% 8.654 94.414 0.16% 0.008µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sql/obfuscate_sql_string execution_time [74.329µs; 74.362µs] or [-0.022%; +0.022%] None None None

Group 10

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 11db3f1 1723042292 ban/stabilize-criterion-results
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time 555.591µs 556.345µs ± 1.216µs 556.240µs ± 0.159µs 556.420µs 556.691µs 556.875µs 573.108µs 3.03% 13.202 179.591 0.22% 0.086µs 1 200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput 1744871.631op/s 1797455.413op/s ± 3818.067op/s 1797786.745op/s ± 513.545op/s 1798262.182op/s 1799018.874op/s 1799302.268op/s 1799884.762op/s 0.12% -13.154 178.708 0.21% 269.978op/s 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time 412.958µs 417.482µs ± 2.309µs 417.288µs ± 1.319µs 418.591µs 421.037µs 422.294µs 435.514µs 4.37% 2.669 17.546 0.55% 0.163µs 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput 2296134.927op/s 2395386.381op/s ± 13065.320op/s 2396426.722op/s ± 7575.286op/s 2404195.596op/s 2412182.767op/s 2414871.720op/s 2421551.253op/s 1.05% -2.472 15.634 0.54% 923.858op/s 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time 196.270µs 201.304µs ± 1.038µs 201.047µs ± 0.334µs 201.548µs 203.096µs 203.696µs 209.933µs 4.42% 2.815 25.086 0.51% 0.073µs 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput 4763414.370op/s 4967732.064op/s ± 25275.428op/s 4973957.406op/s ± 8260.079op/s 4980483.061op/s 4989374.995op/s 4991915.580op/s 5095018.748op/s 2.43% -2.525 23.047 0.51% 1787.243op/s 1 200
normalization/normalize_service/normalize_service/[empty string] execution_time 48.527µs 48.682µs ± 0.053µs 48.691µs ± 0.038µs 48.721µs 48.754µs 48.795µs 48.806µs 0.24% -0.400 -0.072 0.11% 0.004µs 1 200
normalization/normalize_service/normalize_service/[empty string] throughput 20489372.486op/s 20541316.846op/s ± 22287.375op/s 20537858.724op/s ± 15954.029op/s 20557155.329op/s 20579419.434op/s 20601203.599op/s 20607266.711op/s 0.34% 0.406 -0.066 0.11% 1575.955op/s 1 200
normalization/normalize_service/normalize_service/test_ASCII execution_time 63.121µs 63.560µs ± 0.372µs 63.538µs ± 0.107µs 63.643µs 63.782µs 64.079µs 68.269µs 7.45% 10.157 126.172 0.58% 0.026µs 1 200
normalization/normalize_service/normalize_service/test_ASCII throughput 14647852.643op/s 15733679.764op/s ± 87145.113op/s 15738650.740op/s ± 26557.091op/s 15767259.291op/s 15805775.276op/s 15831434.122op/s 15842603.930op/s 0.66% -9.719 118.901 0.55% 6162.090op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time [556.176µs; 556.513µs] or [-0.030%; +0.030%] None None None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput [1796926.266op/s; 1797984.560op/s] or [-0.029%; +0.029%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time [417.162µs; 417.802µs] or [-0.077%; +0.077%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput [2393575.653op/s; 2397197.108op/s] or [-0.076%; +0.076%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time [201.160µs; 201.448µs] or [-0.071%; +0.071%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput [4964229.132op/s; 4971234.995op/s] or [-0.071%; +0.071%] None None None
normalization/normalize_service/normalize_service/[empty string] execution_time [48.675µs; 48.690µs] or [-0.015%; +0.015%] None None None
normalization/normalize_service/normalize_service/[empty string] throughput [20538228.031op/s; 20544405.662op/s] or [-0.015%; +0.015%] None None None
normalization/normalize_service/normalize_service/test_ASCII execution_time [63.508µs; 63.612µs] or [-0.081%; +0.081%] None None None
normalization/normalize_service/normalize_service/test_ASCII throughput [15721602.290op/s; 15745757.239op/s] or [-0.077%; +0.077%] None None None

Group 11

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 11db3f1 1723042292 ban/stabilize-criterion-results
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
tags/replace_trace_tags execution_time 2.708µs 2.754µs ± 0.016µs 2.754µs ± 0.011µs 2.764µs 2.782µs 2.795µs 2.799µs 1.66% 0.327 -0.109 0.59% 0.001µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
tags/replace_trace_tags execution_time [2.752µs; 2.756µs] or [-0.082%; +0.082%] None None None

Baseline

Omitted due to size.

@pawelchcki pawelchcki mentioned this pull request Aug 6, 2024
Merged
@bantonsson bantonsson force-pushed the ban/stabilize-criterion-results branch 2 times, most recently from 8321493 to ea9fcee Compare August 7, 2024 09:27
@bantonsson bantonsson force-pushed the ban/stabilize-criterion-results branch from ea9fcee to a3c9121 Compare August 7, 2024 09:33
@bantonsson bantonsson changed the title [WIP] Stabilize criterion benchmark results Stabilize criterion benchmark results Aug 7, 2024
@bantonsson bantonsson marked this pull request as ready for review August 7, 2024 10:25
@bantonsson bantonsson requested review from a team as code owners August 7, 2024 10:25
@bantonsson bantonsson force-pushed the ban/stabilize-criterion-results branch from a3c9121 to 11db3f1 Compare August 7, 2024 14:51
@bantonsson bantonsson merged commit 748ea8d into main Aug 7, 2024
34 checks passed
@bantonsson bantonsson deleted the ban/stabilize-criterion-results branch August 7, 2024 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ekump ekump ekump approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@bantonsson @codecov-commenter @ekump