Ica bench #261

oojo12 · 2022-11-03T03:22:37Z

Additions

Adds benchmarking code for linfa-ica it is a small refactor of the provided example. Will add benchmark stats if accepted.

algorithms/linfa-ica/benches/fast_ica.rs

oojo12 · 2022-11-04T20:58:19Z

Example output

     Running benches\fast_ica.rs (target\release\deps\fast_ica-239ba21dab7bb851.exe)
Fast ICA/GFunc_LogCosH/1000
                        time:   [74.308 µs 74.549 µs 74.808 µs]
Found 10 outliers among 100 measurements (10.00%)
  9 (9.00%) high mild
  1 (1.00%) high severe
Fast ICA/GFunc_LogCosH/10000
                        time:   [745.54 µs 747.69 µs 749.94 µs]
Found 10 outliers among 100 measurements (10.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  5 (5.00%) high severe
Fast ICA/GFunc_LogCosH/100000
                        time:   [10.645 ms 10.671 ms 10.700 ms]
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

Fast ICA/GFunc_Cube/1000
                        time:   [90.441 µs 98.776 µs 106.35 µs]
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high mild
Benchmarking Fast ICA/GFunc_Cube/10000: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.7s, enable flat sampling, or reduce sample count to 60.
Fast ICA/GFunc_Cube/10000
                        time:   [1.0323 ms 1.1630 ms 1.2919 ms]
Found 13 outliers among 100 measurements (13.00%)
  10 (10.00%) high mild
  3 (3.00%) high severe
Fast ICA/GFunc_Cube/100000
                        time:   [27.111 ms 27.460 ms 27.798 ms]
Found 9 outliers among 100 measurements (9.00%)
  9 (9.00%) low mild

Fast ICA/GFunc_Exp/1000 time:   [273.59 µs 278.39 µs 284.42 µs]
Found 16 outliers among 100 measurements (16.00%)
  6 (6.00%) low mild
  1 (1.00%) high mild
  9 (9.00%) high severe
Fast ICA/GFunc_Exp/10000
                        time:   [2.8397 ms 2.8547 ms 2.8712 ms]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe
Fast ICA/GFunc_Exp/100000
                        time:   [48.794 ms 49.054 ms 49.330 ms]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

codecov-commenter · 2022-11-04T21:05:49Z

Codecov Report

Base: 39.24% // Head: 38.76% // Decreases project coverage by -0.47% ⚠️

Coverage data is based on head (d6d64a7) compared to base (c52cdb5).
Patch has no changes to coverable lines.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #261      +/-   ##
==========================================
- Coverage   39.24%   38.76%   -0.48%     
==========================================
  Files          92       92              
  Lines        6092     6077      -15     
==========================================
- Hits         2391     2356      -35     
- Misses       3701     3721      +20

Impacted Files	Coverage Δ
algorithms/linfa-ica/src/fast_ica.rs	`51.92% <ø> (-0.97%)`	⬇️
src/metrics_classification.rs	`37.34% <ø> (-2.29%)`	⬇️
algorithms/linfa-linear/src/float.rs	`20.00% <0.00%> (-13.34%)`	⬇️
algorithms/linfa-nn/src/heap_elem.rs	`0.00% <0.00%> (-10.00%)`	⬇️
algorithms/linfa-logistic/src/argmin_param.rs	`41.17% <0.00%> (-8.83%)`	⬇️
...rithms/linfa-clustering/src/k_means/hyperparams.rs	`27.08% <0.00%> (-6.25%)`	⬇️
...linfa-clustering/src/appx_dbscan/cells_grid/mod.rs	`46.42% <0.00%> (-6.21%)`	⬇️
...orithms/linfa-clustering/src/optics/hyperparams.rs	`20.58% <0.00%> (-5.89%)`	⬇️
algorithms/linfa-linear/src/glm/distribution.rs	`54.90% <0.00%> (-5.89%)`	⬇️
algorithms/linfa-svm/src/classification.rs	`42.10% <0.00%> (-5.27%)`	⬇️
... and 52 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

oojo12 · 2022-11-04T23:03:23Z

not sure if the test is flaky but when I run cargo clippy --all-targets -- -D warnings I don't get any errors as reported in the failing codequality Lint

algorithms/linfa-ica/benches/fast_ica.rs

YuhanLiin · 2022-11-05T01:17:08Z

You're not seeing the error because the version of Rust in CI is newer than your local version. Just fix the error so the CI passes.

…bench

oojo12 · 2022-11-05T15:03:29Z

You're not seeing the error because the version of Rust in CI is newer than your local version. Just fix the error so the CI passes.

Gotcha it appears to be a new problem on Master too. Assuming it happened with the Rust release Nov 3rd. I addressed it and reran the test locally with no issues with the latest Rust.

YuhanLiin · 2022-11-05T15:37:51Z

Can you do the benchmark comparison in #228 after merging this?

oojo12 · 2022-11-05T15:45:42Z

Sure thing

oojo12 · 2022-11-05T17:02:10Z

I don't have permission to merge per "Only those with write access to this repository can merge pull requests." However, below are the necessary things that can be dumped in that issue once you merge. Also, the report is attached as a zip file.

Context

Laptop mid charge level with wall charger power plugged in
Laptop not in use during test
Laptop not over heating

Run: cargo bench -p linfa-ica

     Running benches\fast_ica.rs (target\release\deps\fast_ica-d0c3b5752e019610.exe)
Fast ICA/GFunc_Cube/1000
                        time:   [63.292 µs 63.701 µs 64.270 µs]
Found 24 outliers among 100 measurements (24.00%)
  22 (22.00%) high mild
  2 (2.00%) high severe
Fast ICA/GFunc_Cube/10000
                        time:   [594.96 µs 599.06 µs 603.51 µs]
Found 6 outliers among 100 measurements (6.00%)
  5 (5.00%) high mild
  1 (1.00%) high severe
Fast ICA/GFunc_Cube/100000
                        time:   [10.097 ms 10.163 ms 10.236 ms]
Found 8 outliers among 100 measurements (8.00%)
  4 (4.00%) high mild
  4 (4.00%) high severe

Fast ICA/GFunc_Logcosh/1000
                        time:   [80.528 µs 80.737 µs 80.965 µs]
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low severe
  5 (5.00%) low mild
  1 (1.00%) high mild
  1 (1.00%) high severe
Fast ICA/GFunc_Logcosh/10000
                        time:   [790.09 µs 792.99 µs 796.68 µs]
Found 13 outliers among 100 measurements (13.00%)
  1 (1.00%) low mild
  7 (7.00%) high mild
  5 (5.00%) high severe
Fast ICA/GFunc_Logcosh/100000
                        time:   [11.175 ms 11.241 ms 11.311 ms]
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe

Fast ICA/Exp/1000       time:   [95.226 µs 95.656 µs 96.168 µs]
Found 11 outliers among 100 measurements (11.00%)
  6 (6.00%) high mild
  5 (5.00%) high severe
Fast ICA/Exp/10000      time:   [930.76 µs 935.38 µs 941.12 µs]
Found 12 outliers among 100 measurements (12.00%)
  1 (1.00%) low severe
  1 (1.00%) low mild
  7 (7.00%) high mild
  3 (3.00%) high severe
Fast ICA/Exp/100000     time:   [18.221 ms 18.381 ms 18.559 ms]
Found 9 outliers among 100 measurements (9.00%)
  2 (2.00%) high mild
  7 (7.00%) high severe

Run cargo bench -p linfa-ica -F intel-mkl-static

     Running benches\fast_ica.rs (target\release\deps\fast_ica-2e03c80ee39d2cab.exe)
Fast ICA/GFunc_Cube/1000
                        time:   [51.978 µs 52.182 µs 52.431 µs]
                        change: [-35.893% -31.248% -26.694%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 13 outliers among 100 measurements (13.00%)
  9 (9.00%) high mild
  4 (4.00%) high severe
Fast ICA/GFunc_Cube/10000
                        time:   [466.80 µs 469.63 µs 473.14 µs]
                        change: [-21.665% -20.949% -20.232%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  1 (1.00%) low mild
  6 (6.00%) high mild
  2 (2.00%) high severe
Fast ICA/GFunc_Cube/100000
                        time:   [9.2229 ms 9.3005 ms 9.3897 ms]
                        change: [-9.4661% -8.4831% -7.3959%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
  5 (5.00%) high mild
  6 (6.00%) high severe

Fast ICA/GFunc_Logcosh/1000
                        time:   [69.136 µs 69.490 µs 69.884 µs]
                        change: [-13.958% -13.385% -12.795%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) low mild
  4 (4.00%) high mild
  1 (1.00%) high severe
Fast ICA/GFunc_Logcosh/10000
                        time:   [672.37 µs 675.86 µs 680.20 µs]
                        change: [-15.858% -14.933% -13.922%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 20 outliers among 100 measurements (20.00%)
  2 (2.00%) low mild
  4 (4.00%) high mild
  14 (14.00%) high severe
Fast ICA/GFunc_Logcosh/100000
                        time:   [10.314 ms 10.412 ms 10.527 ms]
                        change: [-8.3714% -7.3728% -6.1955%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) high mild
  5 (5.00%) high severe

Fast ICA/Exp/1000       time:   [76.481 µs 76.798 µs 77.180 µs]
                        change: [-20.431% -19.929% -19.446%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 13 outliers among 100 measurements (13.00%)
  4 (4.00%) high mild
  9 (9.00%) high severe
Fast ICA/Exp/10000      time:   [708.61 µs 714.43 µs 721.13 µs]
                        change: [-23.322% -22.474% -21.608%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
Fast ICA/Exp/100000     time:   [16.429 ms 16.604 ms 16.795 ms]
                        change: [-10.997% -9.6641% -8.3325%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  6 (6.00%) high mild
  3 (3.00%) high severe

criterion.zip

oojo12 added 4 commits November 2, 2022 23:08

Create fast_ica.rs

f803ae2

Update Cargo.toml

c5a648a

rustfmt

daac923

fix bench name

29cb501

YuhanLiin reviewed Nov 3, 2022

View reviewed changes

algorithms/linfa-ica/benches/fast_ica.rs Outdated Show resolved Hide resolved

YuhanLiin reviewed Nov 3, 2022

View reviewed changes

algorithms/linfa-ica/benches/fast_ica.rs Outdated Show resolved Hide resolved

YuhanLiin reviewed Nov 3, 2022

View reviewed changes

algorithms/linfa-ica/benches/fast_ica.rs Outdated Show resolved Hide resolved

oojo12 added 3 commits November 4, 2022 16:50

address review

c7c9ca5

add random_seed

3b382fb

add random_seed

ee47d95

cargo fmt

d8c9493

change criterion version

6c18e55

YuhanLiin reviewed Nov 5, 2022

View reviewed changes

algorithms/linfa-ica/benches/fast_ica.rs Outdated Show resolved Hide resolved

YuhanLiin reviewed Nov 5, 2022

View reviewed changes

algorithms/linfa-ica/benches/fast_ica.rs Outdated Show resolved Hide resolved

oojo12 added 3 commits November 5, 2022 10:53

add copy trait to GFunc

46b9629

consolidate

0437dcb

Merge branch 'ica-bench' of https://github.com/oojo12/linfa into ica-…

41cad51

…bench

oojo12 marked this pull request as draft November 5, 2022 14:54

cargo clippy fix

0d7dfcb

oojo12 marked this pull request as ready for review November 5, 2022 15:03

oojo12 added 2 commits November 5, 2022 11:05

rustfmt

11b4074

latest criterion

7407774

YuhanLiin approved these changes Nov 5, 2022

View reviewed changes

Merge branch 'master' into ica-bench

d6d64a7

YuhanLiin merged commit f3f9ee4 into rust-ml:master Nov 5, 2022

oojo12 deleted the ica-bench branch November 5, 2022 18:49

relf mentioned this pull request Jul 10, 2024

use rayon to speed up linfa-logistic #355

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ica bench #261

Ica bench #261

oojo12 commented Nov 3, 2022

oojo12 commented Nov 4, 2022 •

edited

Loading

codecov-commenter commented Nov 4, 2022 •

edited

Loading

oojo12 commented Nov 4, 2022

YuhanLiin commented Nov 5, 2022

oojo12 commented Nov 5, 2022 •

edited

Loading

YuhanLiin commented Nov 5, 2022 •

edited

Loading

oojo12 commented Nov 5, 2022

oojo12 commented Nov 5, 2022

Ica bench #261

Ica bench #261

Conversation

oojo12 commented Nov 3, 2022

Additions

oojo12 commented Nov 4, 2022 • edited Loading

codecov-commenter commented Nov 4, 2022 • edited Loading

Codecov Report

oojo12 commented Nov 4, 2022

YuhanLiin commented Nov 5, 2022

oojo12 commented Nov 5, 2022 • edited Loading

YuhanLiin commented Nov 5, 2022 • edited Loading

oojo12 commented Nov 5, 2022

oojo12 commented Nov 5, 2022

Context

Run: cargo bench -p linfa-ica

Run cargo bench -p linfa-ica -F intel-mkl-static

oojo12 commented Nov 4, 2022 •

edited

Loading

codecov-commenter commented Nov 4, 2022 •

edited

Loading

oojo12 commented Nov 5, 2022 •

edited

Loading

YuhanLiin commented Nov 5, 2022 •

edited

Loading