Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmark comparing against fpdec crate #14

Merged
merged 2 commits into from
Oct 25, 2024

Conversation

MathisWellmann
Copy link
Contributor

@MathisWellmann MathisWellmann commented Oct 25, 2024

Contains benchmarks comparing this crate against fpdec (the previous performance champion AFAIK),
both on primitive operations and real-world uses.

TLDR:
Using const_decimal::Decimal<i32, _> vs fpdec::Decimal for inverse futures pnl calculation gives a staggering improvement of 45x and ~15x better when using i64 datatype.
Also fpdec::Decimal is 32 bytes in size vs 4 (i32) and 8 (i64) respectively for const_decimal::Decimal which is nicer for fast CPU caches.

I added black_box everywhere to avoid any sneaky optimizations by the compiler. Maybe overkill, but idk :)

Example output of cargo bench vs_fpdec show the following on an AMD EPYC 9654 (192) @ 2.400GHz:

     Running benches/vs_fpdec.rs (target/release/deps/vs_fpdec-83de002bf491e535)
Gnuplot not found, using plotters backend
vs_fpdec/fpdec_add      time:   [2.4451 ns 2.4483 ns 2.4523 ns]
                        change: [-1.2607% -0.7954% -0.4444%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 10 outliers among 100 measurements (10.00%)
  2 (2.00%) high mild
  8 (8.00%) high severe
vs_fpdec/const_decimal_i32_add
                        time:   [814.50 ps 814.84 ps 815.20 ps]
                        change: [-0.0961% -0.0533% -0.0154%] (p = 0.01 < 0.05)
                        Change within noise threshold.
Found 11 outliers among 100 measurements (11.00%)
  3 (3.00%) high mild
  8 (8.00%) high severe
vs_fpdec/const_decimal_i64_add
                        time:   [814.17 ps 814.27 ps 814.41 ps]
                        change: [-0.3039% -0.2341% -0.1699%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low mild
  5 (5.00%) high mild
  2 (2.00%) high severe
vs_fpdec/fpdec_sub      time:   [2.4498 ns 2.4500 ns 2.4503 ns]
                        change: [+0.2042% +0.2417% +0.2773%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 12 outliers among 100 measurements (12.00%)
  1 (1.00%) low severe
  4 (4.00%) low mild
  3 (3.00%) high mild
  4 (4.00%) high severe
vs_fpdec/const_decimal_i32_sub
                        time:   [814.23 ps 814.30 ps 814.37 ps]
                        change: [-0.0384% -0.0197% -0.0016%] (p = 0.04 < 0.05)
                        Change within noise threshold.
Found 6 outliers among 100 measurements (6.00%)
  6 (6.00%) high mild
vs_fpdec/const_decimal_i64_sub
                        time:   [814.40 ps 814.82 ps 815.39 ps]
                        change: [+0.0303% +0.2686% +0.6741%] (p = 0.16 > 0.05)
                        No change in performance detected.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) high mild
  5 (5.00%) high severe
vs_fpdec/fpdec_mul      time:   [5.9715 ns 5.9720 ns 5.9726 ns]
                        change: [-0.0592% -0.0017% +0.0367%] (p = 0.94 > 0.05)
                        No change in performance detected.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  2 (2.00%) high severe
vs_fpdec/const_decimal_i32_mul
                        time:   [1.8998 ns 1.8999 ns 1.9001 ns]
                        change: [-0.0379% -0.0257% -0.0143%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild
  2 (2.00%) high severe
vs_fpdec/const_decimal_i64_mul
                        time:   [4.8865 ns 4.8868 ns 4.8871 ns]
                        change: [-0.7091% -0.6783% -0.6476%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) low mild
  4 (4.00%) high mild
  2 (2.00%) high severe
vs_fpdec/fpdec_div      time:   [20.875 ns 20.893 ns 20.911 ns]
                        change: [-0.0012% +0.0896% +0.1809%] (p = 0.05 > 0.05)
                        No change in performance detected.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) low mild
  1 (1.00%) high mild
vs_fpdec/const_decimal_i32_div
                        time:   [1.9108 ns 1.9127 ns 1.9148 ns]
                        change: [+0.5683% +0.6867% +0.8264%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild
vs_fpdec/const_decimal_i64_div
                        time:   [4.8990 ns 4.9027 ns 4.9065 ns]
                        change: [+0.1999% +0.2549% +0.3122%] (p = 0.00 < 0.05)
                        Change within noise threshold.
vs_fpdec/fpdec_linear_futures_pnl
                        time:   [11.676 ns 11.677 ns 11.678 ns]
                        change: [+0.0726% +0.0960% +0.1268%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 6 outliers among 100 measurements (6.00%)
  4 (4.00%) low mild
  1 (1.00%) high mild
  1 (1.00%) high severe
vs_fpdec/const_decimal_i32_linear_futures_pnl
                        time:   [3.2695 ns 3.2700 ns 3.2705 ns]
                        change: [+0.2395% +0.3555% +0.4685%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 11 outliers among 100 measurements (11.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  6 (6.00%) high severe
vs_fpdec/const_decimal_i64_linear_futures_pnl
                        time:   [9.5042 ns 9.5049 ns 9.5057 ns]
                        change: [+0.1193% +0.1339% +0.1489%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe
vs_fpdec/fpdec_inverse_futures_pnl
                        time:   [148.56 ns 148.59 ns 148.62 ns]
                        change: [+0.0942% +0.1194% +0.1442%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 9 outliers among 100 measurements (9.00%)
  7 (7.00%) high mild
  2 (2.00%) high severe
vs_fpdec/const_decimal_i32_inverse_futures_pnl
                        time:   [3.2717 ns 3.2733 ns 3.2748 ns]
                        change: [+0.5274% +0.8624% +1.4825%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe
vs_fpdec/const_decimal_i64_inverse_futures_pnl
                        time:   [10.054 ns 10.055 ns 10.055 ns]
                        change: [+0.1513% +0.1756% +0.1995%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) low mild
  5 (5.00%) high mild
  1 (1.00%) high severe

@OliverNChalk OliverNChalk merged commit cd8c1d3 into OliverNChalk:main Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants