Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tame benchmark time #280

Open
vtjnash opened this issue Feb 25, 2021 · 1 comment
Open

tame benchmark time #280

vtjnash opened this issue Feb 25, 2021 · 1 comment

Comments

@vtjnash
Copy link
Member

vtjnash commented Feb 25, 2021

The tests take too long, here's why:

$ grep '^[^ ]' 093b2a6ea943f4c70fef4742453e73dd1aba255c_primary.out 
RUNNING BENCHMARKS...
(1/20) benchmarking "shootout"...
done (took 70.756991683 seconds)
(2/20) benchmarking "string"...
done (took 90.13968044 seconds)
(3/20) benchmarking "linalg"...
done (took 833.003473297 seconds)
(4/20) benchmarking "parallel"...
done (took 17.074202133 seconds)
(5/20) benchmarking "find"...
done (took 139.282557616 seconds)
(6/20) benchmarking "tuple"...
done (took 107.72978849 seconds)
(7/20) benchmarking "dates"...
done (took 112.767580891 seconds)
(8/20) benchmarking "micro"...
done (took 43.087218279 seconds)
(9/20) benchmarking "io"...
done (took 61.65084442 seconds)
(10/20) benchmarking "scalar"... # not unreasonable tests, but takes 2 seconds each
done (took 2877.452812717 seconds)
(11/20) benchmarking "sparse"... # "matmul" has over 200 combinations, probably could be more "sparse"
done (took 1278.73521643 seconds)
(12/20) benchmarking "broadcast"...
done (took 104.842238879 seconds)
(13/20) benchmarking "union"... # 341 tests may be reasonable, but taking 3 seconds each
done (took 1192.683332499 seconds)
(14/20) benchmarking "simd"...
done (took 337.777793678 seconds)
(15/20) benchmarking "random"...
done (took 619.551763952 seconds)
(16/20) benchmarking "problem"...
done (took 172.609170893 seconds)
(17/20) benchmarking "array"... # 608 "index" tests may be a bit much, and take 4 seconds each
done (took 2795.986356153 seconds)
(18/20) benchmarking "misc"...
done (took 158.855686545 seconds)
(19/20) benchmarking "sort"...
done (took 648.501236383 seconds)
(20/20) benchmarking "collection"...
done (took 916.806169471 seconds)
@KristofferC
Copy link
Contributor

How about turning of the GC scrubbing (via the gctrial and gcsample in https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/doc/manual.md#benchmark-parameters)? We only use the minimum value anyway for all reports and analyses.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants