Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Makefile to hyperfine benchmarks #2533

Merged
merged 2 commits into from
Dec 1, 2023
Merged

Add Makefile to hyperfine benchmarks #2533

merged 2 commits into from
Dec 1, 2023

Conversation

jonaprieto
Copy link
Collaborator

@jonaprieto jonaprieto commented Nov 22, 2023

This PR adds a target for our Makefile to run benchmarks using hyperfine.
We run different commands which can be easily modified in bench/hyperfine/Makefile.

make hyperfine-benchmarks

After running this on your terminal, checkout the generated file bench/hyperfine/README.md for the results.

The results below are the runs using the following module.

module fibo;

import Stdlib.Prelude open;

fib : Nat → Nat → Nat → Nat
  | zero x1 _ := x1
  | (suc n) x1 x2 := fib n x2 (x1 + x2);

fibonacci (n : Nat) : Nat := fib n 0 1;

main : IO := printNatLn (fibonacci 50);

(For now, I got the following on my machine (ran in about 5min or less)
Darwin Jonathans-MacBook-Pro-2.local 22.6.0 Darwin Kernel Version 22.6.0: Fri Sep 15 13:41:28 PDT 2023; root:xnu-8796.141.3.700.8~1/RELEASE_ARM64_T6020 arm64 arm)

  • The binary without the version below is the latest commit on main.

Hyperfine Benchmarks

dev parse

Command Mean [ms] Min [ms] Max [ms] Relative
juvix-v0.4.3 dev parse fibo.juvix 184.3 ± 5.4 178.5 193.2 1.06 ± 0.04
juvix-v0.5.0 dev parse fibo.juvix 184.7 ± 6.9 179.2 201.3 1.06 ± 0.05
juvix-v0.5.1 dev parse fibo.juvix 174.2 ± 5.4 167.9 181.0 1.00
juvix-v0.5.2 dev parse fibo.juvix 181.3 ± 1.6 179.5 184.1 1.04 ± 0.03
juvix-v0.5.3 dev parse fibo.juvix 1185.8 ± 5.7 1178.4 1197.3 6.81 ± 0.21
juvix-v0.5.4 dev parse fibo.juvix 1308.4 ± 6.9 1297.6 1319.3 7.51 ± 0.23
juvix dev parse fibo.juvix 1311.0 ± 5.5 1303.2 1318.5 7.53 ± 0.23

dev highlight

Command Mean [ms] Min [ms] Max [ms] Relative
juvix-v0.4.3 dev highlight fibo.juvix 770.2 ± 67.5 742.9 961.6 1.01 ± 0.09
juvix-v0.5.0 dev highlight fibo.juvix 762.7 ± 11.8 749.9 787.2 1.00
juvix-v0.5.1 dev highlight fibo.juvix 849.3 ± 9.3 836.5 863.9 1.11 ± 0.02
juvix-v0.5.2 dev highlight fibo.juvix 873.5 ± 21.1 855.2 918.9 1.15 ± 0.03
juvix-v0.5.3 dev highlight fibo.juvix 2035.9 ± 69.5 1946.9 2125.4 2.67 ± 0.10
juvix-v0.5.4 dev highlight fibo.juvix 2218.0 ± 57.3 2169.5 2316.4 2.91 ± 0.09
juvix dev highlight fibo.juvix 2206.5 ± 55.9 2150.4 2296.6 2.89 ± 0.09

typecheck

Command Mean [ms] Min [ms] Max [ms] Relative
juvix-v0.4.3 typecheck fibo.juvix 684.1 ± 5.1 674.1 691.0 1.00
juvix-v0.5.0 typecheck fibo.juvix 695.9 ± 14.2 679.0 720.6 1.02 ± 0.02
juvix-v0.5.1 typecheck fibo.juvix 808.5 ± 18.2 780.8 848.6 1.18 ± 0.03
juvix-v0.5.2 typecheck fibo.juvix 816.0 ± 13.9 802.3 846.3 1.19 ± 0.02
juvix-v0.5.3 typecheck fibo.juvix 1934.3 ± 29.0 1907.3 1992.1 2.83 ± 0.05
juvix-v0.5.4 typecheck fibo.juvix 2106.8 ± 19.6 2075.6 2138.9 3.08 ± 0.04
juvix typecheck fibo.juvix 2135.3 ± 29.9 2107.4 2183.2 3.12 ± 0.05

compile -o /dev/null

Command Mean [ms] Min [ms] Max [ms] Relative
juvix-v0.4.3 compile -o /dev/null fibo.juvix 754.3 ± 4.9 745.2 759.8 1.00 ± 0.01
juvix-v0.5.0 compile -o /dev/null fibo.juvix 752.6 ± 3.9 745.2 757.6 1.00
juvix-v0.5.1 compile -o /dev/null fibo.juvix 845.4 ± 5.7 837.6 857.0 1.12 ± 0.01
juvix-v0.5.2 compile -o /dev/null fibo.juvix 883.7 ± 15.7 864.5 918.3 1.17 ± 0.02
juvix-v0.5.3 compile -o /dev/null fibo.juvix 1990.8 ± 12.5 1975.0 2010.4 2.65 ± 0.02
juvix-v0.5.4 compile -o /dev/null fibo.juvix 2193.9 ± 5.6 2182.7 2200.5 2.91 ± 0.02
juvix compile -o /dev/null fibo.juvix 2197.4 ± 11.6 2185.0 2226.0 2.92 ± 0.02

eval

Command Mean [ms] Min [ms] Max [ms] Relative
juvix-v0.4.3 eval fibo.juvix 680.9 ± 5.4 669.7 687.1 1.00
juvix-v0.5.0 eval fibo.juvix 681.7 ± 5.0 675.0 689.8 1.00 ± 0.01
juvix-v0.5.1 eval fibo.juvix 772.7 ± 2.7 769.0 777.9 1.13 ± 0.01
juvix-v0.5.2 eval fibo.juvix 796.1 ± 2.7 791.9 799.5 1.17 ± 0.01
juvix-v0.5.3 eval fibo.juvix 1902.2 ± 6.5 1889.6 1913.2 2.79 ± 0.02
juvix-v0.5.4 eval fibo.juvix 2101.3 ± 5.9 2093.1 2112.1 3.09 ± 0.03
juvix eval fibo.juvix 2111.1 ± 17.4 2085.5 2148.3 3.10 ± 0.04

@jonaprieto jonaprieto marked this pull request as ready for review December 1, 2023 16:02
@jonaprieto jonaprieto modified the milestones: 0.5.6, 0.5.5 Dec 1, 2023
@jonaprieto jonaprieto requested a review from paulcadman December 1, 2023 16:07
Makefile Outdated Show resolved Hide resolved
Co-authored-by: Paul Cadman <git@paulcadman.dev>
@jonaprieto jonaprieto merged commit b8a016f into main Dec 1, 2023
4 checks passed
@jonaprieto jonaprieto deleted the hyperfine-tests branch December 1, 2023 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants