Skip to content

Commit

Permalink
1.0.14
Browse files Browse the repository at this point in the history
  • Loading branch information
@liborty
liborty committed Oct 25, 2023
1 parent 624f262 commit 0e1a495
Show file tree
Hide file tree
Showing 4 changed files with 85 additions and 109 deletions.
4 changes: 2 additions & 2 deletions Cargo.toml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "times"
version = "1.0.13"
version = "1.0.14"
authors = ["Libor Spacek"]
edition = "2021"
description = "Timing Benchmark, Repeated Runs, with Statistics"
Expand All @@ -22,4 +22,4 @@ maintenance = { status = "actively-developed" }
[dependencies]
indxvec = "1.8"
ran = "1.1"
medians = "2.3"
medians = "3.0"
12 changes: 7 additions & 5 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,28 +13,28 @@ use times::*;

## Introduction

This crate will be typically added in `Cargo.toml` under `[dev-dependecies]` and then used by source files under `tests` or `benches` directories. It can be used whenever the runtime speed comparisons are of interest, which is practically always.
This crate will be typically added in `Cargo.toml` under `[dev-dependecies]` and then used by source files in `tests` or `benches` directories. It can be used whenever the runtime speed comparisons are of interest, which is practically always.

`Times` is suitable for testing algorithms that work on a whole `Vec` of data, for example sort. Or even whole matrices of data `&[Vec<T>]`.

The correctness of the results
should be tested separately. Here the results produced by the algorithms are thrown away and only the execution time in nanoseconds is recorded.

Random data are automatically generated using `ran` crate and then algorithms from a given array of closures are repeatedly run and their statistics are collected (median of the execution *times* and the measurements standard error). Repeated runs reduce the temporary effects of changing machine loads. The effects of outliers are minimised by using `mad` instead of standard deviation. Mad stand for median of absolute differences from median; it is the most stable measure of data spread.
Random data are automatically generated using `ran` crate and then algorithms from a given array of closures are repeatedly executed and their statistics are collected (median of the execution *times* and their spread (`mad`). Mad stand for median of absolute differences from median; it is the most stable measure of data spread. Repeated runs reduce the inevitable temporary effects of changing machine loads, cache utilisation, etc. The effects of outliers are minimised by using `mad` instead of standard deviation.

All the algorithms are run over the same data for exact comparisons but the data is changed for each repeat run.

The error estimates the doubt about the reliability of repeated measurements. High value means poor reliability. Relative measurement accuracy can be often increased by increasing the number of repeats. The extraneous influence of the machine load is also reduced as the length of the data vectors increases.
The spread expresses the doubt about the reliability of repeated measurements. High value means poor reliability. Relative measurement inaccuracy (spread as percentage) can be often decreased by increasing the number of repeats. The extraneous influence of the machine load is also reduced as the length of the data vectors increases.

We generate new random data for each repeated run. The differences in errors between the algorithms inform us about their relative stability under changing data. Some algorithms suffer from data sensitivity (poor worst-case performance) and this may be indicated by relatively high errors, e.g. for `rust-sort` (the standard Rust sort).
We generate new random data for each repeated run. The differences in spreads between the algorithms inform us about their relative stability under changing data. Some algorithms suffer from data sensitivity (poor worst-case performance) and this may be indicated by relatively high spreads, e.g. for `rust-sort` (the standard Rust sort).

The tests are also automatically repeated over different lengths of the input data vectors, in specified range and step. This enables comparisons of algorithms as the difficulty of the problem increases. The algorithms with lower computational complexity and/or faster implementations will start to win more convincingly at greater lengths.

When the data length becomes too large, then the process may have to be externally terminated. Depending, of course, on the algorithms and the speed of the machine. It is recommended to use modest values at first.

## Main Features

* Mad, as more stable standard error.
* Mad, as more stable measure of spread (measurement error).

* Ease of Use - just specify:
* the type of the random test data,
Expand Down Expand Up @@ -68,6 +68,8 @@ Please see [`tests/test.rs`](https://github.com/liborty/times/blob/main/tests/te

## Appendix - Recent Releases

**Version 1.0.14** Upgraded to Medians v 3.0.0, enabled checking for Nans, improved reports.

**Version 1.0.13** Fixed some dependency problems in tests.

**Version 1.0.12** Removed dependency on `devtimer`.
Expand Down
149 changes: 66 additions & 83 deletions src/lib.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,19 @@ use medians::Medianf64;
use ran::{generators::get_seed, *};

fn report(names: &[&str], meds: &[f64], stderrs: &[f64]) {
let medsx = meds.mergesort_indexed();
let medsx = meds.isort_indexed(0..meds.len(),|a:&f64,b| a.total_cmp(b));
let meds_sorted = medsx.unindex(meds, true);
let names_sorted = medsx.unindex(names, true);
let stderrs_sorted = medsx.unindex(stderrs, true);
for i in 0..names.len() {
println!(
"{MG}{:<18}{GR}{:>13.0} ± {:>7.0}{UN}",
names_sorted[i], meds_sorted[i], stderrs_sorted[i]
);
}
"{YL}{:<18}{GR}{:>13.0} ±{:>7.0} ~{:>5.2}%{YL} {:>7.4}{UN}",
names_sorted[i],
meds_sorted[i],
stderrs_sorted[i],
100.0*stderrs_sorted[i]/meds_sorted[i],
meds_sorted[i]/meds_sorted[0]);
};
}
fn heading(data:&str,c1:usize,c2:usize,step:usize,rows:usize,repeats:usize) {
println!(
Expand All @@ -43,10 +46,10 @@ pub fn bench(repeats: usize, names: &[&str], closures: &[fn()]) {
let now = Instant::now(); // = UNIX_EPOCH.elapsed().unwrap().as_nanos() as u64;timer.start();
closure();
times.push(now.elapsed().as_nanos() as f64);
}
let medmad = times.medstats().expect("bench mestats");
meds.push(medmad.centre);
stderrs.push(medmad.dispersion);
};
let med = times.medf_checked().expect("bench Nan detected");
meds.push(med);
stderrs.push(times.madf(med));
}
report(names, &meds, &stderrs);
}
Expand Down Expand Up @@ -79,12 +82,10 @@ pub fn mutbenchu8(
let now = Instant::now();
closure(&mut data);
times.push(now.elapsed().as_nanos() as f64);
}
let medmad = times
.medstats()
.expect("mutbenchu8 medstats");
meds.push(medmad.centre);
stderrs.push(medmad.dispersion);
};
let med = times.medf_checked().expect("mutbenchu8 Nan detected");
meds.push(med);
stderrs.push(times.madf(med));
}
report(names, &meds, &stderrs);
}
Expand Down Expand Up @@ -118,12 +119,10 @@ pub fn mutbenchu16(
let now = Instant::now();
closure(&mut data);
times.push(now.elapsed().as_nanos() as f64);
}
let medmad = times
.medstats()
.expect("mutbenchu8 medstats");
meds.push(medmad.centre);
stderrs.push(medmad.dispersion);
};
let med = times.medf_checked().expect("mutbenchu16 Nan detected");
meds.push(med);
stderrs.push(times.madf(med));
}
report(names, &meds, &stderrs);
}
Expand Down Expand Up @@ -157,12 +156,10 @@ pub fn mutbenchu64(
let now = Instant::now();
closure(&mut data);
times.push(now.elapsed().as_nanos() as f64);
}
let medmad = times
.medstats()
.expect("mutbenchu64 medstats");
meds.push(medmad.centre);
stderrs.push(medmad.dispersion);
};
let med = times.medf_checked().expect("mutbenchu64 Nan detected");
meds.push(med);
stderrs.push(times.madf(med));
}
report(names, &meds, &stderrs);
}
Expand Down Expand Up @@ -196,12 +193,10 @@ pub fn mutbenchf64(
let now = Instant::now();
closure(&mut data);
times.push(now.elapsed().as_nanos() as f64);
}
let medmad = times
.medstats()
.expect("mutbenchf64 medstats");
meds.push(medmad.centre);
stderrs.push(medmad.dispersion);
};
let med = times.medf_checked().expect("mutbenchf64 Nan detected");
meds.push(med);
stderrs.push(times.madf(med));
}
report(names, &meds, &stderrs);
}
Expand Down Expand Up @@ -235,10 +230,10 @@ pub fn benchu8(
let now = Instant::now();
closure(&data);
times.push(now.elapsed().as_nanos() as f64);
}
let medmad = times.medstats().expect("benchu8 medstats");
meds.push(medmad.centre);
stderrs.push(medmad.dispersion);
};
let med = times.medf_checked().expect("benchu8 Nan detected");
meds.push(med);
stderrs.push(times.madf(med));
}
report(names, &meds, &stderrs);
}
Expand Down Expand Up @@ -272,10 +267,10 @@ pub fn benchu16(
let now = Instant::now();
closure(&data);
times.push(now.elapsed().as_nanos() as f64);
}
let medmad = times.medstats().expect("benchu8 medstats");
meds.push(medmad.centre);
stderrs.push(medmad.dispersion);
};
let med = times.medf_checked().expect("benchu16 Nan detected");
meds.push(med);
stderrs.push(times.madf(med));
}
report(names, &meds, &stderrs);
}
Expand Down Expand Up @@ -309,13 +304,11 @@ pub fn benchu64(
let now = Instant::now();
closure(&data);
times.push(now.elapsed().as_nanos() as f64);
}
let medmad = times
.medstats()
.expect("benchu64 medstats");
meds.push(medmad.centre);
stderrs.push(medmad.dispersion);
}
};
let med = times.medf_checked().expect("benchu64 Nan detected");
meds.push(med);
stderrs.push(times.madf(med));
};
report(names, &meds, &stderrs);
}
}
Expand Down Expand Up @@ -348,13 +341,11 @@ pub fn benchf64(
let now = Instant::now();
closure(&data);
times.push(now.elapsed().as_nanos() as f64);
}
let medmad = times
.medstats()
.expect("benchf64 medstats");
meds.push(medmad.centre);
stderrs.push(medmad.dispersion);
}
};
let med = times.medf_checked().expect("benchf64 Nan detected");
meds.push(med);
stderrs.push(times.madf(med));
};
report(names, &meds, &stderrs);
}
}
Expand Down Expand Up @@ -388,13 +379,11 @@ pub fn benchvvu8(
let now = Instant::now();
closure(&data);
times.push(now.elapsed().as_nanos() as f64);
}
let medmad = times
.medstats()
.expect("benchvvu8 medstats");
meds.push(medmad.centre);
stderrs.push(medmad.dispersion);
}
};
let med = times.medf_checked().expect("benchvvu8 Nan detected");
meds.push(med);
stderrs.push(times.madf(med));
};
report(names, &meds, &stderrs);
}
}
Expand Down Expand Up @@ -428,13 +417,11 @@ pub fn benchvvu16(
let now = Instant::now();
closure(&data);
times.push(now.elapsed().as_nanos() as f64);
}
let medmad = times
.medstats()
.expect("benchvvu8 medstats");
meds.push(medmad.centre);
stderrs.push(medmad.dispersion);
}
};
let med = times.medf_checked().expect("benchvvu16 Nan detected");
meds.push(med);
stderrs.push(times.madf(med));
};
report(names, &meds, &stderrs);
}
}
Expand Down Expand Up @@ -468,13 +455,11 @@ pub fn benchvvf64(
let now = Instant::now();
closure(&data);
times.push(now.elapsed().as_nanos() as f64);
}
let medmad = times
.medstats()
.expect("benchvvf64 medstats");
meds.push(medmad.centre);
stderrs.push(medmad.dispersion);
}
};
let med = times.medf_checked().expect("benchvvf64 Nan detected");
meds.push(med);
stderrs.push(times.madf(med));
};
report(names, &meds, &stderrs);
}
}
Expand Down Expand Up @@ -508,13 +493,11 @@ pub fn benchvvu64(
let now = Instant::now();
closure(&data);
times.push(now.elapsed().as_nanos() as f64);
}
let medmad = times
.medstats()
.expect("benchvvu64 medstats");
meds.push(medmad.centre);
stderrs.push(medmad.dispersion);
}
};
let med = times.medf_checked().expect("benchvvu64 Nan detected");
meds.push(med);
stderrs.push(times.madf(med));
};
report(names, &meds, &stderrs);
}
}
29 changes: 10 additions & 19 deletions tests/tests.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,31 +11,22 @@ use times::{bench,mutbenchu8,mutbenchu16,mutbenchu64,mutbenchf64};

#[test]
fn benchtests() {
const NAMES:[&str;6] = [ "merge-sort","hash-sort","mergesort_indexed","hashsort_indexed","muthashsort","rust-sort" ];
const NAMES:[&str;3] = [ "muthashsort","rust-sort","mutisort" ];

const CLOSURESU8:[fn(&mut[u8]);6] = [
|v:&mut[_]| { v.sortm(true); },
|v:&mut[_]| { v.sorth(|t:&u8| *t as f64, true); },
|v:&mut[_]| { v.mergesort_indexed(); },
|v:&mut[_]| { v.hashsort_indexed(|t:&u8| *t as f64); },
const CLOSURESU8:[fn(&mut[u8]);3] = [
|v:&mut[_]| { v.muthashsort(|t:&u8| *t as f64); },
|v:&mut[_]| { v.sort_unstable(); } ];
|v:&mut[_]| { v.sort_unstable(); },
|v:&mut[_]| { v.mutisort(0..v.len(),|a,b| a.cmp(b)); } ];

const CLOSURESU16:[fn(&mut[u16]);6] = [
|v:&mut[_]| { v.sortm(true); },
|v:&mut[_]| { v.sorth(|t:&u16| *t as f64, true); },
|v:&mut[_]| { v.mergesort_indexed(); },
|v:&mut[_]| { v.hashsort_indexed(|t:&u16| *t as f64); },
const CLOSURESU16:[fn(&mut[u16]);3] = [
|v:&mut[_]| { v.muthashsort(|t:&u16| *t as f64); },
|v:&mut[_]| { v.sort_unstable(); } ];
|v:&mut[_]| { v.sort_unstable(); },
|v:&mut[_]| { v.mutisort(0..v.len(),|a,b| a.cmp(b)); } ];

const CLOSURESF64:[fn(&mut[f64]);6] = [
|v:&mut[_]| { v.sortm(true); },
|v:&mut[_]| { v.sorth(|t:&f64| *t, true); },
|v:&mut[_]| { v.mergesort_indexed(); },
|v:&mut[_]| { v.hashsort_indexed(|t:&f64| *t); },
const CLOSURESF64:[fn(&mut[f64]);3] = [
|v:&mut[_]| { v.muthashsort(|t:&f64| *t); },
|v:&mut[_]| { v.sort_unstable_by(|a,b| a.total_cmp(b)); } ];
|v:&mut[_]| { v.sort_unstable_by(|a,b| a.total_cmp(b)); },
|v:&mut[_]| { v.mutisort(0..v.len(),|a,b| a.total_cmp(b)); } ];

set_seeds(0); // intialise random numbers generator
// Rnum encapsulates the type of the data items
Expand Down

0 comments on commit 0e1a495

Please sign in to comment.