Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: libaustin #136

Draft
wants to merge 22 commits into
base: devel
Choose a base branch
from
Draft

feat: libaustin #136

wants to merge 22 commits into from

Conversation

P403n1x87
Copy link
Owner

Requirements for Adding, Changing, Fixing or Removing a Feature

Fill out the template below. Any pull request that does not include enough
information to be reviewed in a timely manner may be closed at the maintainers'
discretion.

Description of the Change

Alternate Designs

Regressions

Verification Process

@P403n1x87 P403n1x87 added this to the libaustin milestone Oct 20, 2022
@P403n1x87 P403n1x87 self-assigned this Oct 20, 2022
@P403n1x87 P403n1x87 changed the base branch from master to devel October 20, 2022 16:07
@P403n1x87 P403n1x87 changed the title Libaustin feat: libaustin Oct 20, 2022
@P403n1x87 P403n1x87 force-pushed the libaustin branch 2 times, most recently from 5b7b1e2 to 68133de Compare October 20, 2022 16:18
@codecov
Copy link

codecov bot commented Oct 20, 2022

Codecov Report

Patch coverage: 44.32% and project coverage change: -2.49 ⚠️

Comparison is base (0484829) 71.49% compared to head (a7a292b) 69.00%.

Additional details and impacted files
@@            Coverage Diff             @@
##            devel     #136      +/-   ##
==========================================
- Coverage   71.49%   69.00%   -2.49%     
==========================================
  Files          25       25              
  Lines        2473     2568      +95     
  Branches      730      733       +3     
==========================================
+ Hits         1768     1772       +4     
- Misses        381      461      +80     
- Partials      324      335      +11     
Impacted Files Coverage Δ
src/linux/py_proc.h 59.27% <ø> (ø)
src/logging.c 73.46% <ø> (-8.54%) ⬇️
src/stack.h 68.75% <0.00%> (-9.38%) ⬇️
src/stats.c 82.22% <ø> (-6.67%) ⬇️
src/py_proc.c 63.34% <35.18%> (-6.97%) ⬇️
src/austin.c 68.12% <56.66%> (-1.23%) ⬇️
src/py_thread.c 73.80% <70.00%> (-3.14%) ⬇️

... and 5 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

P403n1x87 and others added 9 commits February 26, 2023 11:41
This change improves the report from benchmark runs by adding t-tests
for each scenario. This allows the benchmarks to provide a summary at
the end with the scenarios that are likely to show perfomance difference
between the latest released version and the one that comes with the PR
the benchmarks run for.

This also updates the versions that are being benchmarked to include the
latest releases.
ci(benchmarks): improve benchmark reporting
This change adds a data validation workflow. Data is validated by
performing a Hotelling T2 test on the collected data.
This change adds extra steps to the CI workflows to build Python
wheels so that the Austin binary can be installed with pip from
PyPI.
The d_name variable is a table of 256. It can not be null.
@P403n1x87 P403n1x87 force-pushed the libaustin branch 8 times, most recently from 159584a to 4704f2f Compare May 18, 2023 21:39
@github-actions
Copy link

github-actions bot commented May 18, 2023

Austin Benchmarks

Running Austin benchmarks with Python 3.10.11

Wall time [sampling interval: 1]

Wall time [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 120000 ± 2000 1 ± 0 1.7e-05 ± 1e-05 13 ± 0
3.5.0 118000 ± 5000 1 ± 0 2.1e-05 ± 7e-06 13.2 ± 0.6
dev 117000 ± 8000 1 ± 0 0.0001 ± 0.0001 13 ± 1
Wall time [sampling interval: 10]

Wall time [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 110000 ± 10000 0.545 ± 0.002 1.6e-05 ± 7e-06 14 ± 2
3.5.0 115000 ± 4000 0.545 ± 0.003 1.6e-05 ± 4e-06 13.4 ± 0.5
dev 115000 ± 5000 0.544 ± 0.002 1.8e-05 ± 5e-06 13.2 ± 0.4
Wall time [sampling interval: 100]

Wall time [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 10500 ± 500 0.0006 ± 0.0002 6e-05 ± 4e-05 14.2 ± 0.4
3.5.0 10600 ± 300 0.0007 ± 0.0003 6e-05 ± 3e-05 14.5 ± 0.5
dev 10600 ± 500 0.0008 ± 0.0006 5e-05 ± 4e-05 14.3 ± 0.5
Wall time [sampling interval: 1000]

Wall time [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 1830 ± 40 0 ± 0 0.0 ± 0.0001 17.1 ± 0.6
3.5.0 1810 ± 60 0.0001 ± 0.0002 0.0 ± 0.0001 18 ± 1
dev 1860 ± 20 0.0001 ± 0.0003 0.0 ± 0.0001 17.6 ± 0.7
CPU time [sampling interval: 1]

CPU time [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 51000 ± 4000 1 ± 0 3e-05 ± 2e-05 23 ± 1
3.5.0 53000 ± 2000 1 ± 0 3e-05 ± 2e-05 22.1 ± 0.6
dev 53000 ± 3000 1 ± 0 2e-05 ± 2e-05 22.3 ± 0.7
CPU time [sampling interval: 10]

CPU time [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 51000 ± 6000 0.9964 ± 0.0008 3e-05 ± 2e-05 23 ± 2
3.5.0 52000 ± 3000 0.9964 ± 0.0002 2e-05 ± 1e-05 22.6 ± 0.5
dev 53000 ± 3000 0.9959 ± 0.0003 3e-05 ± 1e-05 22.3 ± 0.8
CPU time [sampling interval: 100]

CPU time [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 5400 ± 200 0.0009 ± 0.0002 0.00014 ± 0.0001 22.3 ± 0.5
3.5.0 5320 ± 80 0.001 ± 0.0002 0.0001 ± 0.0001 22.8 ± 0.6
dev 5400 ± 100 0.001 ± 0.0004 0.0001 ± 0.0001 22.2 ± 0.6
CPU time [sampling interval: 1000]

CPU time [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 950 ± 3 0 ± 0 0.0004 ± 0.0005 26.1 ± 0.3
3.5.0 950 ± 4 0.0001 ± 0.0002 0.0003 ± 0.0004 26.1 ± 0.3
dev 952 ± 4 0.0001 ± 0.0005 0.0001 ± 0.0003 26 ± 2
RSA keygen [sampling interval: 1]

RSA keygen [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 40000 ± 1000 1 ± 0 0.00017 ± 5e-05 24.2 ± 0.8
3.5.0 40000 ± 1000 1 ± 0 0.00012 ± 6e-05 24.1 ± 0.7
dev 39800 ± 800 1 ± 0 0.0002 ± 0.0002 24.3 ± 0.5
RSA keygen [sampling interval: 10]

RSA keygen [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 41000 ± 1000 0.99 ± 0.01 0.0002 ± 0.0002 23.9 ± 0.7
3.5.0 40000 ± 1000 0.98 ± 0.02 0.0003 ± 0.0004 24.0 ± 0.9
dev 40000 ± 900 0.98 ± 0.02 0.0002 ± 0.0002 24.2 ± 0.6
RSA keygen [sampling interval: 100]

RSA keygen [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 6240 ± 30 0.003 ± 0.003 0.001 ± 0.0008 30 ± 2
3.5.0 6290 ± 30 0.002 ± 0.001 0.0007 ± 0.0005 28 ± 2
dev 6230 ± 10 0.0013 ± 0.0009 0.0007 ± 0.0008 30 ± 2
RSA keygen [sampling interval: 1000]

RSA keygen [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 942 ± 2 0 ± 0 0.001 ± 0.002 33 ± 2
3.5.0 942 ± 3 0 ± 0 0.0005 ± 0.0009 31 ± 1
dev 943 ± 1 0 ± 0 0.0003 ± 0.0007 32 ± 2
Full metrics [sampling interval: 1]

Full metrics [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 50000 ± 3000 1 ± 0 4e-05 ± 2e-05 31 ± 2
3.5.0 51000 ± 1000 1 ± 0 4e-05 ± 2e-05 29.6 ± 0.7
dev 51000 ± 2000 1 ± 0 4e-05 ± 2e-05 30 ± 1
Full metrics [sampling interval: 10]

Full metrics [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 51000 ± 3000 1 ± 0 6e-05 ± 2e-05 30 ± 2
3.5.0 52000 ± 1000 1 ± 0 4e-05 ± 2e-05 30 ± 1
dev 52000 ± 2000 1 ± 0 5e-05 ± 1e-05 30 ± 2
Full metrics [sampling interval: 100]

Full metrics [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 10100 ± 500 0.0017 ± 0.0004 0.0001 ± 5e-05 32.4 ± 0.8
3.5.0 10000 ± 200 0.0016 ± 0.0003 9e-05 ± 6e-05 32.6 ± 0.8
dev 9800 ± 300 0.0016 ± 0.0005 8e-05 ± 6e-05 33 ± 1
Full metrics [sampling interval: 1000]

Full metrics [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 1850 ± 20 0.0002 ± 0.0004 0.0001 ± 0.0002 39 ± 1
3.5.0 1860 ± 20 0.0002 ± 0.0003 0.0001 ± 0.0002 38 ± 1
dev 1840 ± 20 0.0 ± 0.0001 0.0002 ± 0.0002 36.7 ± 0.5
Multiprocess wall time [sampling interval: 1]

Multiprocess wall time [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 2900 ± 200 1 ± 0 0.00013 ± 4e-05 320 ± 20
3.5.0 3400 ± 100 1 ± 0 0.00013 ± 2e-05 280 ± 10
dev 3330 ± 50 1 ± 0 0.00016 ± 5e-05 289 ± 4
Multiprocess wall time [sampling interval: 10]

Multiprocess wall time [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 3000 ± 200 1 ± 0 0.00014 ± 5e-05 310 ± 20
3.5.0 3500 ± 200 1 ± 0 0.00016 ± 5e-05 280 ± 10
dev 3300 ± 200 1 ± 0 0.00015 ± 5e-05 290 ± 10
Multiprocess wall time [sampling interval: 100]

Multiprocess wall time [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 2700 ± 200 0.04 ± 0.02 0.00012 ± 4e-05 340 ± 20
3.5.0 3300 ± 200 0.04 ± 0.02 0.00011 ± 4e-05 290 ± 20
dev 3200 ± 100 0.055 ± 0.006 8e-05 ± 3e-05 300 ± 10
Multiprocess wall time [sampling interval: 1000]

Multiprocess wall time [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 2700 ± 200 0.013 ± 0.004 3e-05 ± 3e-05 120 ± 10
3.5.0 3100 ± 200 0.006 ± 0.003 2e-05 ± 2e-05 60 ± 10
dev 3000 ± 200 0.006 ± 0.003 3e-05 ± 2e-05 60 ± 10

Benchmark Summary

Comparison of dev against 3.5.0.

The following scenarios show a statistically significant difference in performance between the two versions.

Sample Rate Saturation Error Rate Sampling Speed
CPU time [sampling interval: 10] 🟡 🟢 🟡 🟡
CPU time [sampling interval: 100] 🟡 🟡 🟡 🟢
RSA keygen [sampling interval: 100] 🔴 🟡 🟡 🔴
Full metrics [sampling interval: 1000] 🔴 🟡 🟡 🟢
Multiprocess wall time [sampling interval: 1] 🔴 🟡 🟡 🔴

@P403n1x87 P403n1x87 force-pushed the libaustin branch 5 times, most recently from 33f27b5 to 17e7249 Compare May 21, 2023 15:16
@P403n1x87 P403n1x87 force-pushed the libaustin branch 2 times, most recently from 5e45ba6 to 403d2e6 Compare May 21, 2023 16:17
P403n1x87 and others added 12 commits May 30, 2023 13:43
Use the more correct expression for computing the lasti index with
Python 3.11.
ci(ubuntu): update before install
The version of cppcheck available from the package manager tends to be
outdated. In this change we build from sources instead. We cache the
build result for faster execution.
chore: minor compilation fixes with gcc 11.3
ci(checks): build cppcheck from sources
refactor: use proper lasti formula for 3.11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants