Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci(benchmarks): improve benchmark reporting #174

Merged
merged 1 commit into from
Feb 28, 2023
Merged

ci(benchmarks): improve benchmark reporting #174

merged 1 commit into from
Feb 28, 2023

Conversation

P403n1x87
Copy link
Owner

@P403n1x87 P403n1x87 commented Feb 26, 2023 •
edited
Loading

This change improves the report from benchmark runs by adding t-tests for each scenario. This allows the benchmarks to provide a summary at the end with the scenarios likely to show performance differences between the latest released version and the one that comes with the PR the benchmarks run for.

This also updates the versions being benchmarked to include the latest releases.

@github-actions
Copy link

github-actions bot commented Feb 26, 2023 •
edited
Loading

Austin Benchmarks

Running Austin benchmarks with Python 3.10.10

Wall time [sampling interval: 1]

Wall time [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 101000 ± 4000 1 ± 0 9e-06 ± 5e-06 15.4 ± 0.7
3.5.0 102000 ± 5000 1 ± 0 1.6e-05 ± 6e-06 15.4 ± 0.8
dev 105000 ± 3000 1 ± 0 1e-05 ± 7e-06 15.1 ± 0.3
Wall time [sampling interval: 10]

Wall time [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 98000 ± 6000 0.576 ± 0.003 1.1e-05 ± 7e-06 16 ± 1
3.5.0 97000 ± 6000 0.575 ± 0.004 1e-05 ± 6e-06 16 ± 1
dev 97000 ± 7000 0.573 ± 0.002 1.3e-05 ± 9e-06 16 ± 1
Wall time [sampling interval: 100]

Wall time [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 9900 ± 200 0.0036 ± 0.0006 5e-05 ± 3e-05 18.7 ± 0.8
3.5.0 9900 ± 300 0.0038 ± 0.0008 4e-05 ± 5e-05 18.8 ± 0.6
dev 9900 ± 200 0.004 ± 0.001 5e-05 ± 5e-05 18.9 ± 0.7
Wall time [sampling interval: 1000]

Wall time [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 1780 ± 20 0.0004 ± 0.0005 3e-05 ± 9e-05 31 ± 1
3.5.0 1780 ± 20 0.0003 ± 0.0003 0.0001 ± 0.0002 31 ± 2
dev 1770 ± 20 0.0007 ± 0.0005 0 ± 0 31 ± 2
CPU time [sampling interval: 1]

CPU time [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 42000 ± 2000 1 ± 0 3e-05 ± 1e-05 28.4 ± 1.0
3.5.0 42000 ± 3000 1 ± 0 3e-05 ± 1e-05 28 ± 1
dev 41000 ± 2000 1 ± 0 2e-05 ± 1e-05 28.5 ± 0.8
CPU time [sampling interval: 10]

CPU time [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 41000 ± 4000 0.9995 ± 0.0007 1e-05 ± 1e-05 29 ± 2
3.5.0 42000 ± 2000 0.9997 ± 0.0004 3e-05 ± 2e-05 28.8 ± 0.8
dev 42000 ± 2000 0.9999 ± 0.0002 2e-05 ± 1e-05 28.2 ± 0.9
CPU time [sampling interval: 100]

CPU time [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 4800 ± 200 0.011 ± 0.002 0.00011 ± 7e-05 36 ± 1
3.5.0 4700 ± 100 0.011 ± 0.002 0.0001 ± 0.0001 36 ± 2
dev 4690 ± 70 0.011 ± 0.001 0.00011 ± 0.0001 36 ± 1
CPU time [sampling interval: 1000]

CPU time [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 913 ± 4 0.001 ± 0.002 0.0003 ± 0.0003 55 ± 4
3.5.0 912 ± 6 0.0008 ± 0.0009 0.0002 ± 0.0004 56 ± 4
dev 910 ± 6 0.0007 ± 0.0009 0.0002 ± 0.0004 56 ± 4
RSA keygen [sampling interval: 1]

RSA keygen [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 30000 ± 2000 1 ± 0 0.0003 ± 0.0002 32 ± 1
3.5.0 30000 ± 3000 1 ± 0 0.0004 ± 0.0003 32 ± 2
dev 31000 ± 2000 1 ± 0 0.0003 ± 0.0005 31 ± 2
RSA keygen [sampling interval: 10]

RSA keygen [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 31000 ± 2000 0.998 ± 0.002 0.0003 ± 0.0002 32 ± 2
3.5.0 30000 ± 2000 0.9996 ± 0.0006 0.0004 ± 0.0005 32 ± 1
dev 31000 ± 700 0.996 ± 0.004 0.0003 ± 0.0003 31.4 ± 0.7
RSA keygen [sampling interval: 100]

RSA keygen [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 6000 ± 100 0.03 ± 0.01 0.0004 ± 0.0003 44 ± 5
3.5.0 5990 ± 70 0.02 ± 0.02 0.0004 ± 0.0003 42 ± 5
dev 6040 ± 70 0.016 ± 0.007 0.0006 ± 0.0004 40 ± 3
RSA keygen [sampling interval: 1000]

RSA keygen [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 904 ± 4 0.0 ± 0.001 0.002 ± 0.002 70 ± 6
3.5.0 901 ± 5 0.0007 ± 0.0008 0.002 ± 0.002 68 ± 5
dev 902 ± 3 0.0005 ± 0.0008 0.001 ± 0.001 66 ± 3
Full metrics [sampling interval: 1]

Full metrics [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 40000 ± 900 1 ± 0 4e-05 ± 2e-05 38 ± 1
3.5.0 39000 ± 1000 1 ± 0 3e-05 ± 1e-05 39 ± 1
dev 40800 ± 700 1 ± 0 3e-05 ± 2e-05 37.4 ± 0.8
Full metrics [sampling interval: 10]

Full metrics [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 38000 ± 4000 1 ± 0 3e-05 ± 2e-05 41 ± 4
3.5.0 39000 ± 2000 1 ± 0 3e-05 ± 1e-05 40 ± 3
dev 39000 ± 2000 1 ± 0 3e-05 ± 2e-05 40 ± 2
Full metrics [sampling interval: 100]

Full metrics [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 8200 ± 300 0.034 ± 0.002 0.0001 ± 9e-05 56 ± 2
3.5.0 8200 ± 200 0.032 ± 0.004 0.0001 ± 8e-05 54 ± 2
dev 8500 ± 400 0.032 ± 0.004 0.001 ± 0.001 56 ± 4
Full metrics [sampling interval: 1000]

Full metrics [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 1790 ± 20 0.001 ± 0.0008 0.0001 ± 0.0001 72 ± 4
3.5.0 1780 ± 20 0.0011 ± 0.0007 0.0002 ± 0.0003 74 ± 2
dev 1780 ± 30 0.001 ± 0.0008 0.0001 ± 0.0002 73 ± 4
Multiprocess wall time [sampling interval: 1]

Multiprocess wall time [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 1800 ± 200 1 ± 0 0.00017 ± 4e-05 340 ± 10
3.5.0 2400 ± 100 1 ± 0 0.00017 ± 4e-05 400 ± 20
dev 2300 ± 200 1 ± 0 0.00018 ± 8e-05 400 ± 30
Multiprocess wall time [sampling interval: 10]

Multiprocess wall time [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 1790 ± 90 1 ± 0 0.00022 ± 6e-05 350 ± 20
3.5.0 2400 ± 100 1 ± 0 0.0002 ± 0.0002 400 ± 20
dev 2400 ± 100 1 ± 0 0.00014 ± 6e-05 410 ± 20
Multiprocess wall time [sampling interval: 100]

Multiprocess wall time [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 1800 ± 200 0.04 ± 0.02 0.00013 ± 5e-05 360 ± 20
3.5.0 2300 ± 100 0.06 ± 0.03 0.00012 ± 5e-05 420 ± 20
dev 2300 ± 100 0.07 ± 0.01 9e-05 ± 5e-05 410 ± 20
Multiprocess wall time [sampling interval: 1000]

Multiprocess wall time [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 1510 ± 60 0.007 ± 0.006 4e-05 ± 4e-05 90 ± 30
3.5.0 2110 ± 90 0.028 ± 0.008 4e-05 ± 2e-05 190 ± 40
dev 2140 ± 90 0.025 ± 0.009 3e-05 ± 1e-05 170 ± 40

Benchmark Summary

Comparison of dev against 3.5.0.

The following scenarios show a statistically significant difference in performance between the two versions.

Sample Rate Saturation Error Rate Sampling Speed
RSA keygen [sampling interval: 10] 🟢 🟢 🟡 🟢
Full metrics [sampling interval: 1] 🟢 🟡 🟡 🟢

@codecov
Copy link

codecov bot commented Feb 26, 2023 •
edited
Loading

Codecov Report

Base: 70.61% // Head: 70.53% // Decreases project coverage by -0.09% ⚠️

Coverage data is based on head (b9d38aa) compared to base (01a247a).
Patch has no changes to coverable lines.

Additional details and impacted files 
@@            Coverage Diff             @@
##            devel     #174      +/-   ##
==========================================
- Coverage   70.61%   70.53%   -0.09%     
==========================================
  Files          25       25              
  Lines        2474     2474              
  Branches      730      730              
==========================================
- Hits         1747     1745       -2     
- Misses        402      403       +1     
- Partials      325      326       +1
Impacted Files Coverage Δ
src/py_proc_list.c 74.79% <0.00%> (-4.88%) ⬇️
src/austin.c 68.34% <0.00%> (-1.01%) ⬇️
src/linux/py_proc.h 59.27% <0.00%> (-0.61%) ⬇️
src/py_proc.c 65.62% <0.00%> (-0.45%) ⬇️
src/py_thread.c 76.53% <0.00%> (ø)
src/stack.h 81.25% <0.00%> (+3.12%) ⬆️
src/py_string.h 78.20% <0.00%> (+11.53%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@P403n1x87 P403n1x87 force-pushed the ci/benchmarks-ttest branch 3 times, most recently from a1b01d9 to d503987 Compare February 28, 2023 00:48
@P403n1x87
ci(benchmarks): improve benchmark reporting
b9d38aa 
This change improves the report from benchmark runs by adding t-tests
for each scenario. This allows the benchmarks to provide a summary at
the end with the scenarios that are likely to show perfomance difference
between the latest released version and the one that comes with the PR
the benchmarks run for.

This also updates the versions that are being benchmarked to include the
latest releases.
@P403n1x87 P403n1x87 marked this pull request as ready for review February 28, 2023 10:51
@P403n1x87 P403n1x87 merged commit 4c3e26b into devel Feb 28, 2023
@P403n1x87 P403n1x87 deleted the ci/benchmarks-ttest branch March 17, 2023 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
ci/cd
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@P403n1x87