Table of Content

Overview
- Measurements
Test Cases
- Brainfuck
  - bench.b
  - mandel.b
- Base64
- Json
- Matmul
- Primes
Tests Execution
Contribution

Overview

The benchmarks follow the criteria:

They are written as the average software developer would write them, i.e.
- The algorithms are implemented as cited in public sources;
- The libraries are used as described in the tutorials, documentation and examples;
- The used data structures are idiomatic.
The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.
All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks: jit-benchmarks, crystal-benchmarks-game

Measurements

The measured values are:

time spent for the benchmark execution (loading required data and code self-testing are not measured);
memory consumption of the benchmark process, reported as base + increase, where base is the RSS before the benchmark and increase is the peak increase of the RSS during the benchmark;
energy consumption of the CPU package during the benchmark: PP0 (cores) + PP1 (uncores like GPU) + DRAM. Currently, only Intel CPU are supported via the powercap interface.

All values are presented as: median_{±median absolute deviation}.

UPDATE: 2022-05-30

Test Cases

Brainfuck

Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:

Verbose (default). Prints the output immediately.
Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

Brainfuck

bench.b

Language	Time, s	Memory, MiB	Energy, J
Racket (Syntax Objects)	1.294_±0.001	104.83_±00.98 + 0.00_±00.00	47.02_±00.02
C++/g++	1.317_±0.000	1.65_±00.00 + 0.00_±00.00	52.77_±00.08
Rust	1.589_±0.001	2.12_±00.03 + 0.00_±00.00	63.57_±00.08
D/ldc2	1.655_±0.000	3.21_±00.03 + 0.00_±00.00	66.25_±00.07
Java	1.655_±0.001	37.33_±00.46 + 0.90_±00.37	64.58_±00.05
D/gdc	1.660_±0.000	7.49_±00.02 + 0.00_±00.00	67.55_±00.71
C/clang	1.663_±0.000	0.69_±00.03 + 0.00_±00.00	64.54_±00.06
C/gcc	1.667_±0.000	0.71_±00.00 + 0.00_±00.00	64.71_±00.06
Kotlin/JVM	1.677_±0.006	40.74_±00.11 + 0.42_±00.16	65.50_±00.18
C++/clang++	1.691_±0.000	1.48_±00.01 + 0.00_±00.00	64.32_±00.73
Zig	1.747_±0.000	0.70_±00.00 + 0.00_±00.00	68.20_±00.38
F#/.NET Core	1.896_±0.000	37.86_±00.09 + 0.54_±00.00	76.35_±00.41
C#/.NET Core	1.902_±0.000	33.85_±00.29 + 0.00_±00.00	76.27_±00.19
Go	1.904_±0.000	2.79_±00.08 + 0.00_±00.00	72.13_±00.49
OCaml	1.912_±0.000	2.71_±00.02 + 2.30_±00.03	85.13_±00.36
Chez Scheme	1.923_±0.001	24.91_±00.08 + 4.40_±00.02	79.48_±00.05
Nim/gcc	1.976_±0.000	2.02_±00.02 + 0.00_±00.00	78.82_±00.74
Racket	2.012_±0.010	93.21_±00.51 + 0.00_±00.00	81.10_±00.77
Vala/gcc	2.090_±0.001	4.25_±00.05 + 0.00_±00.00	77.06_±00.85
Nim/clang	2.100_±0.002	2.43_±00.05 + 0.00_±00.00	81.52_±00.15
Go/gccgo	2.172_±0.000	24.55_±00.10 + 0.00_±00.00	86.43_±00.27
Vala/clang	2.262_±0.000	4.26_±00.07 + 0.00_±00.00	82.35_±00.19
V/gcc	2.340_±0.000	0.66_±00.00 + 0.00_±00.00	87.35_±00.63
Crystal	2.384_±0.000	3.46_±00.01 + 0.00_±00.00	95.11_±00.44
MLton	2.608_±0.000	0.80_±00.03 + 1.01_±00.03	102.87_±00.28
C#/Mono	2.685_±0.001	24.99_±00.13 + 0.00_±00.00	107.58_±00.08
V/clang	2.967_±0.019	0.66_±00.00 + 0.00_±00.00	115.67_±00.92
Haskell (MArray)	3.165_±0.001	4.18_±00.04 + 3.97_±00.00	131.78_±00.77
Scala	3.222_±0.023	66.15_±00.25 + 141.06_±00.32	133.08_±00.82
D/dmd	3.267_±0.000	3.76_±00.01 + 0.00_±00.00	122.89_±00.09
Julia	3.565_±0.001	208.86_±00.13 + 0.99_±00.03	134.35_±00.18
Node.js	4.016_±0.002	37.68_±00.02 + 0.85_±00.08	161.03_±00.35
Swift	5.415_±0.000	13.76_±00.14 + 0.00_±00.00	200.89_±02.89
Lua/luajit	6.492_±0.008	2.41_±00.04 + 0.00_±00.00	259.77_±00.54
Ruby/truffleruby	7.352_±0.037	291.61_±00.77 + 396.22_±08.35	347.59_±02.47
Ruby/truffleruby (--jvm)	8.767_±0.078	376.87_±16.30 + 458.33_±29.58	430.85_±06.14
Python/pypy	12.207_±0.065	65.14_±00.02 + 29.65_±00.03	512.74_±00.94
Haskell	13.368_±0.019	4.31_±00.04 + 3.97_±00.00	595.87_±04.15
Ruby (--jit)	39.635_±0.156	270.95_±00.02 + 0.02_±00.00	1618.92_±08.59
Elixir	43.913_±0.156	70.50_±00.34 + 0.00_±00.00	2011.98_±10.43
Lua	50.617_±0.073	2.30_±00.02 + 0.00_±00.00	1941.25_±23.16
Ruby/jruby	78.965_±0.799	183.82_±00.75 + 83.58_±00.83	3500.74_±39.03
Ruby	85.918_±0.486	14.41_±00.03 + 0.00_±00.00	3615.58_±32.78
Python	185.860_±1.077	10.63_±00.00 + 0.00_±00.00	7415.67_±57.06
Tcl (FP)	239.705_±1.971	4.56_±00.03 + 0.00_±00.00	10232.99_±120.09
Perl	324.279_±3.019	7.06_±00.05 + 0.00_±00.00	12967.03_±121.22
Tcl (OOP)	492.812_±2.162	4.55_±00.08 + 0.00_±00.00	20801.83_±165.07

mandel.b

Mandel in Brainfuck

Language	Time, s	Memory, MiB	Energy, J
C++/g++	10.461_±0.014	1.65_±00.00 + 2.34_±00.05	429.24_±01.97
Racket (Syntax Objects)	14.090_±0.116	105.24_±00.45 + 72.57_±00.64	563.20_±06.84
C/gcc	14.203_±0.011	0.68_±00.03 + 1.05_±00.02	561.03_±02.60
Kotlin/JVM	14.337_±0.030	40.77_±00.09 + 1.00_±00.14	580.64_±03.91
V/gcc	14.519_±0.017	0.72_±00.00 + 2.08_±00.05	589.11_±05.55
C++/clang++	14.656_±0.009	1.50_±00.02 + 2.09_±00.02	588.07_±01.37
Rust	14.823_±0.008	2.17_±00.04 + 0.25_±00.00	593.32_±00.71
D/ldc2	14.864_±0.019	3.20_±00.03 + 0.77_±00.00	598.33_±02.87
D/gdc	15.138_±0.015	7.47_±00.03 + 0.77_±00.00	640.93_±01.15
C/clang	15.431_±0.010	0.69_±00.03 + 1.05_±00.04	644.66_±01.57
Zig	16.222_±0.009	0.70_±00.00 + 1.61_±00.01	669.85_±04.37
Go	17.065_±0.259	3.34_±00.11 + 1.26_±00.00	674.90_±08.80
Swift	18.068_±0.088	14.62_±00.06 + 0.00_±00.00	734.16_±06.70
Java	18.283_±0.012	37.81_±00.06 + 1.29_±00.13	732.18_±00.83
Vala/gcc	18.498_±0.004	4.09_±00.05 + 2.04_±00.02	694.48_±01.06
C#/.NET Core	18.529_±0.008	34.13_±00.07 + 0.83_±00.01	751.34_±04.79
Crystal	19.010_±0.215	3.46_±00.01 + 0.38_±00.02	778.36_±08.01
Nim/clang	19.309_±0.305	2.43_±00.04 + 0.57_±00.00	784.15_±16.13
Vala/clang	19.913_±0.003	4.08_±00.02 + 2.05_±00.06	770.85_±00.76
Go/gccgo	20.039_±0.013	24.59_±00.05 + 1.27_±00.00	845.96_±02.65
Nim/gcc	21.108_±0.569	1.95_±00.01 + 0.51_±00.00	854.87_±21.62
Scala	21.395_±0.016	66.24_±00.05 + 141.19_±00.08	883.73_±05.55
V/clang	22.592_±0.064	0.66_±00.00 + 1.89_±00.03	931.66_±02.36
OCaml	26.203_±0.017	3.48_±00.02 + 8.08_±03.91	1247.53_±12.44
Chez Scheme	26.997_±0.052	25.38_±00.05 + 3.92_±00.01	1180.81_±03.40
C#/Mono	31.357_±0.018	25.04_±00.06 + 0.83_±00.00	1331.72_±01.53
Node.js	31.704_±0.292	37.65_±00.06 + 5.37_±00.02	1317.45_±17.77
Lua/luajit	35.127_±0.069	2.42_±00.01 + 0.50_±00.00	1408.43_±04.07
Racket	36.927_±0.067	92.66_±00.27 + 0.00_±00.00	1636.94_±08.74
D/dmd	37.471_±0.015	3.76_±00.01 + 0.77_±00.00	1377.33_±08.82
F#/.NET Core	38.358_±0.052	37.99_±00.04 + 2.08_±00.00	1587.43_±03.20
MLton	44.833_±0.049	1.59_±00.04 + 4.11_±00.00	1945.45_±09.66
Python/pypy	48.373_±0.180	64.76_±00.06 + 30.07_±00.03	2068.90_±24.04
Haskell (MArray)	49.631_±0.622	4.12_±00.02 + 5.83_±00.00	2120.11_±20.32
Julia	64.532_±0.078	209.49_±00.10 + 0.97_±00.08	2486.31_±13.12
Ruby/truffleruby (--jvm)	78.552_±1.319	398.37_±13.36 + 440.08_±26.60	3295.51_±37.63
Ruby/truffleruby	83.372_±0.453	291.66_±00.26 + 251.66_±23.69	3567.05_±19.14
Haskell	186.042_±2.425	4.34_±00.04 + 29.24_±00.00	8142.94_±109.17

Base64

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Base64

Language	Time, s	Memory, MiB	Energy, J
C/clang (aklomp)	0.100_±0.001	2.04_±00.04 + 0.00_±00.00	4.73_±00.07
C/gcc (aklomp)	0.101_±0.000	2.09_±00.02 + 0.00_±00.00	4.80_±00.05
Rust	0.969_±0.000	2.58_±00.03 + 0.01_±00.00	39.56_±00.22
V/clang	0.995_±0.001	1.91_±00.02 + 0.00_±00.00	37.66_±00.14
C/clang	0.997_±0.000	2.01_±00.04 + 0.00_±00.00	37.14_±00.36
C/gcc	1.012_±0.000	2.05_±00.04 + 0.00_±00.00	37.95_±00.09
Nim/clang	1.024_±0.001	2.82_±00.03 + 4.44_±00.00	41.08_±00.18
D/ldc2	1.074_±0.003	3.58_±00.03 + 3.58_±00.00	45.02_±00.50
Nim/gcc	1.081_±0.002	2.31_±00.03 + 4.44_±00.06	42.48_±00.56
V/gcc	1.092_±0.000	1.63_±00.01 + 0.44_±00.04	42.03_±00.17
Crystal	1.154_±0.002	3.95_±00.01 + 1.17_±00.02	51.11_±00.15
Ruby (--jit)	1.326_±0.001	271.35_±00.02 + 66.67_±00.01	52.24_±00.23
Ruby	1.336_±0.002	14.83_±00.01 + 58.68_±00.04	52.41_±00.27
Java	1.511_±0.008	38.37_±00.06 + 258.46_±28.01	61.39_±00.49
Scala	1.562_±0.005	64.37_±00.18 + 315.74_±04.03	67.26_±01.06
Vala/gcc	1.563_±0.001	5.49_±00.03 + 0.44_±00.08	61.08_±00.51
Vala/clang	1.563_±0.001	5.60_±00.04 + 0.30_±00.03	60.93_±00.56
Kotlin/JVM	1.635_±0.007	40.46_±00.12 + 263.10_±18.51	69.46_±01.12
C++/clang++ (libcrypto)	1.725_±0.005	4.46_±00.05 + 0.68_±00.06	68.58_±00.43
C++/g++ (libcrypto)	1.726_±0.004	5.12_±00.09 + 0.68_±00.06	68.44_±00.24
Perl (MIME::Base64)	1.779_±0.014	14.86_±00.04 + 0.07_±00.00	70.08_±00.39
Go	1.852_±0.001	4.07_±00.06 + 4.57_±00.38	78.91_±00.38
PHP	2.015_±0.022	17.09_±00.09 + 0.00_±00.00	81.94_±00.93
Python	2.115_±0.002	10.41_±00.06 + 0.18_±00.00	78.44_±00.28
Node.js	2.234_±0.003	36.96_±00.03 + 39.88_±00.01	90.04_±00.67
D/gdc	2.645_±0.001	7.62_±00.02 + 3.91_±00.00	114.17_±00.89
D/dmd	2.736_±0.001	4.04_±00.02 + 3.36_±00.00	118.22_±00.86
Zig	3.062_±0.001	1.47_±00.01 + 0.31_±00.00	122.30_±00.82
F#/.NET Core	3.130_±0.065	38.63_±00.03 + 18.63_±01.03	117.75_±01.18
Python/pypy	3.580_±0.002	65.49_±00.10 + 30.62_±00.15	155.88_±01.19
Go/gccgo	3.716_±0.007	25.42_±00.09 + 7.60_±00.21	167.47_±00.66
C#/.NET Core	3.748_±0.081	34.49_±00.06 + 20.71_±03.36	137.05_±01.81
Tcl	3.993_±0.005	5.09_±00.02 + 0.16_±00.00	160.32_±00.37
C#/Mono	4.698_±0.019	25.75_±00.07 + 18.71_±00.09	194.78_±00.66
Ruby/truffleruby (--jvm)	5.090_±0.008	343.44_±03.17 + 242.66_±37.11	247.93_±01.44
Julia	5.095_±0.007	228.19_±00.26 + 62.22_±00.04	190.73_±02.14
Ruby/jruby	10.597_±0.088	182.58_±00.94 + 77.53_±03.02	422.40_±02.95
Perl (MIME::Base64::Perl)	13.988_±0.057	16.16_±00.09 + 0.22_±00.03	575.68_±03.73
Ruby/truffleruby	17.438_±0.022	281.42_±00.63 + 235.47_±06.16	701.31_±05.07

Json

Testing parsing and simple calculating of values from a big JSON file.

Few notes:

gason mutates input strings;
simdjson requires input strings with batch of trailing zeros: a special zero padding for SIMD instructions;
DAW JSON Link "NoCheck" skips some JSON structure correctness checks;
DAW JSON Link, gason, default (not "Precise") RapidJSON, and D implementations except Mir-based have some inaccuracies in number parsing:

Json

Language	Time, s	Memory, MiB	Energy, J
C++/g++ (simdjson On-Demand)	0.067_±0.000	113.45_±00.09 + 59.81_±00.00	2.78_±00.03
C++/clang++ (simdjson On-Demand)	0.068_±0.000	112.55_±00.07 + 60.36_±00.06	2.83_±00.01
C++/clang++ (DAW JSON Link NoCheck)	0.077_±0.000	112.44_±00.03 + 0.00_±00.00	3.14_±00.01
C++/g++ (DAW JSON Link NoCheck)	0.079_±0.000	113.19_±00.03 + 0.00_±00.00	3.16_±00.02
C++/g++ (DAW JSON Link)	0.087_±0.000	113.22_±00.07 + 0.00_±00.00	3.58_±00.03
C++/clang++ (DAW JSON Link)	0.089_±0.000	112.44_±00.02 + 0.00_±00.00	3.67_±00.04
Rust (Serde Custom)	0.107_±0.000	111.99_±00.05 + 0.00_±00.00	4.56_±00.02
Rust (Serde Typed)	0.112_±0.000	111.97_±00.07 + 12.03_±00.00	4.72_±00.01
C++/g++ (gason)	0.133_±0.000	113.14_±00.05 + 96.80_±00.06	5.28_±00.03
C++/clang++ (simdjson DOM)	0.136_±0.001	112.48_±00.03 + 177.15_±00.06	5.79_±00.03
C++/g++ (simdjson DOM)	0.138_±0.000	113.01_±00.09 + 176.39_±00.19	5.88_±00.01
D/ldc2 (Mir Asdf DOM)	0.146_±0.000	112.73_±00.04 + 61.27_±00.03	6.09_±00.02
C++/clang++ (gason)	0.149_±0.000	112.48_±00.03 + 96.97_±00.06	6.02_±00.01
C++/g++ (RapidJSON)	0.167_±0.000	113.26_±00.05 + 126.54_±00.42	6.93_±00.04
C++/clang++ (RapidJSON)	0.221_±0.000	112.45_±00.00 + 129.02_±00.04	8.98_±00.04
D/ldc2 (Mir Amazon's Ion DOM)	0.231_±0.000	112.72_±00.05 + 80.80_±00.00	9.50_±00.02
C++/g++ (RapidJSON Precise)	0.246_±0.000	113.24_±00.03 + 128.82_±00.06	10.41_±00.10
C++/g++ (Boost.JSON)	0.394_±0.000	113.22_±00.03 + 435.88_±00.04	16.44_±00.13
Nim/clang (jsony)	0.399_±0.000	112.16_±00.02 + 42.34_±00.03	16.95_±00.22
C++/g++ (RapidJSON SAX)	0.404_±0.001	112.96_±00.04 + 0.00_±00.00	16.68_±00.07
C++/clang++ (RapidJSON Precise)	0.410_±0.000	112.47_±00.03 + 129.01_±00.02	16.63_±00.05
C++/clang++ (Boost.JSON)	0.413_±0.000	112.44_±00.00 + 436.25_±00.06	17.23_±00.20
Nim/gcc (jsony)	0.422_±0.000	111.72_±00.03 + 42.28_±00.00	18.09_±00.12
C++/g++ (RapidJSON SAX Precise)	0.450_±0.001	112.96_±00.01 + 0.00_±00.00	19.34_±00.09
C++/clang++ (RapidJSON SAX)	0.496_±0.000	194.69_±00.00 + 0.00_±00.00	19.89_±00.05
Go (jsoniter)	0.519_±0.002	230.86_±00.02 + 1.07_±00.04	21.80_±00.07
Node.js	0.548_±0.003	147.87_±00.07 + 188.65_±00.28	25.00_±00.14
Java (DSL-JSON)	0.621_±0.015	258.17_±00.07 + 138.46_±00.77	32.18_±00.99
Rust (Serde Untyped)	0.633_±0.001	111.90_±00.03 + 840.05_±00.03	25.50_±00.07
V/clang	0.635_±0.001	111.23_±00.05 + 496.21_±00.00	26.63_±00.29
V/gcc	0.635_±0.001	111.25_±00.02 + 496.12_±00.03	26.51_±00.25
C++/clang++ (RapidJSON SAX Precise)	0.658_±0.001	194.71_±00.03 + 0.00_±00.00	26.74_±00.12
Python/pypy	0.678_±0.001	285.59_±00.03 + 124.88_±00.00	29.23_±00.13
Crystal (Pull)	0.688_±0.002	113.86_±00.02 + 18.18_±00.02	30.53_±00.29
Crystal (Schema)	0.694_±0.002	113.84_±00.00 + 48.57_±00.04	30.58_±00.36
Julia (JSON3)	0.744_±0.004	378.93_±00.36 + 378.85_±00.20	30.54_±00.32
C#/.NET Core (System.Text.Json)	0.759_±0.002	479.31_±00.03 + 138.58_±00.03	33.20_±00.31
Zig	0.769_±0.001	110.75_±00.01 + 12.18_±00.00	30.36_±00.06
Perl (Cpanel::JSON::XS)	0.833_±0.005	125.42_±00.05 + 402.78_±00.06	34.78_±00.29
Go	0.855_±0.001	117.10_±00.06 + 79.79_±00.11	35.57_±00.01
Nim/clang (Packedjson)	0.929_±0.000	112.50_±00.02 + 294.16_±00.00	38.21_±00.10
Crystal	0.937_±0.010	113.88_±00.01 + 392.00_±00.01	42.07_±00.33
PHP	0.984_±0.001	126.43_±00.10 + 682.01_±00.00	41.13_±00.11
Nim/gcc (Packedjson)	1.010_±0.000	112.00_±00.04 + 294.16_±00.00	41.38_±00.17
C++/clang++ (json-c)	1.203_±0.007	112.72_±00.03 + 1216.17_±00.04	50.96_±00.27
C++/g++ (json-c)	1.208_±0.009	113.28_±00.10 + 1215.96_±00.03	50.54_±00.57
Clojure	1.217_±0.014	450.12_±06.93 + 535.88_±48.40	64.53_±00.76
C++/clang++ (Nlohmann)	1.253_±0.002	112.62_±00.04 + 360.14_±00.02	53.05_±00.28
Go/gccgo	1.263_±0.003	139.06_±00.07 + 83.52_±00.15	52.29_±00.14
C#/.NET Core	1.294_±0.016	487.04_±00.06 + 294.45_±00.04	56.51_±00.63
Nim/clang	1.315_±0.002	112.34_±00.12 + 925.29_±00.10	54.34_±00.27
Nim/gcc	1.349_±0.002	111.95_±00.09 + 919.68_±00.06	56.49_±00.36
CPython (UltraJSON)	1.440_±0.003	121.97_±00.01 + 543.94_±02.33	53.47_±00.22
Ruby (--jit)	1.465_±0.002	381.01_±00.02 + 262.87_±00.01	61.51_±00.41
Ruby	1.471_±0.007	124.42_±00.03 + 262.85_±00.01	62.65_±00.34
C++/g++ (Nlohmann)	1.476_±0.005	113.25_±00.04 + 447.92_±00.04	60.35_±00.21
Python	1.520_±0.003	120.41_±00.01 + 374.96_±00.00	59.77_±00.14
F#/.NET Core (System.Text.Json)	1.802_±0.005	486.98_±00.15 + 384.58_±12.24	79.04_±00.74
Scala (uPickle)	1.849_±0.011	291.27_±00.18 + 665.30_±48.68	90.05_±00.57
Ruby (YAJL)	1.983_±0.025	124.21_±00.01 + 282.40_±00.00	83.42_±01.05
D/ldc2	2.085_±0.003	113.15_±00.02 + 680.09_±00.02	86.06_±00.41
C#/Mono	2.293_±0.021	252.49_±00.14 + 31.58_±00.02	96.48_±01.08
Haskell	2.688_±0.004	115.95_±00.03 + 716.91_±00.26	108.80_±00.08
Rust (jq)	2.791_±0.006	113.65_±00.07 + 778.76_±00.90	115.54_±00.54
Ruby/jruby	3.113_±0.027	442.08_±03.76 + 933.59_±104.90	157.63_±03.53
C++/clang++ (Boost.PropertyTree)	3.126_±0.003	194.87_±00.02 + 1232.87_±00.01	129.76_±00.70
C++/g++ (Boost.PropertyTree)	3.338_±0.006	113.14_±00.03 + 1440.12_±00.02	135.06_±00.77
D/gdc	3.681_±0.009	117.28_±00.03 + 680.72_±00.08	157.65_±00.82
Vala/gcc	3.897_±0.009	114.69_±00.01 + 940.84_±00.03	165.64_±00.87
Vala/clang	4.031_±0.009	114.69_±00.03 + 876.74_±00.02	170.91_±00.96
D/dmd	4.524_±0.007	113.67_±00.02 + 680.09_±00.02	182.17_±00.86
Perl (JSON::Tiny)	9.756_±0.035	125.79_±00.04 + 528.88_±00.06	422.50_±03.92
Ruby/truffleruby (--jvm)	16.470_±0.253	461.41_±25.16 + 2492.61_±30.69	1069.49_±16.47
Ruby/truffleruby	22.345_±0.910	417.31_±01.29 + 2336.16_±85.83	1080.54_±36.02

Matmul

Testing allocating and multiplying matrices.

Matmul

Language	Time, s	Memory, MiB	Energy, J
D/ldc2 (lubeck)	0.046_±0.000	6.57_±00.04 + 54.95_±00.10	4.66_±00.03
Nim/gcc (Arraymancer)	0.066_±0.002	5.52_±00.05 + 54.73_±00.11	5.46_±00.13
C++/clang++ (Eigen)	0.068_±0.003	38.90_±02.59 + 51.51_±02.58	5.53_±00.15
Python (NumPy)	0.070_±0.000	28.41_±00.12 + 54.96_±00.04	6.49_±00.04
C++/g++ (Eigen)	0.070_±0.002	29.55_±08.72 + 60.24_±08.76	5.13_±00.13
Nim/clang (Arraymancer)	0.072_±0.001	6.34_±00.04 + 54.76_±00.08	6.07_±00.07
Java (ND4J)	0.081_±0.005	104.24_±01.83 + 92.25_±00.16	6.35_±00.31
Rust (ndarray)	0.090_±0.000	2.63_±00.04 + 68.48_±00.00	6.00_±00.03
Julia (threads: 4)	0.191_±0.000	235.35_±00.21 + 49.23_±00.90	15.54_±00.02
Julia (threads: 1)	0.510_±0.000	235.65_±00.18 + 49.17_±00.66	21.51_±00.03
Julia (no BLAS)	1.046_±0.010	225.72_±00.30 + 51.62_±00.01	46.15_±00.39
D/ldc2	1.716_±0.001	3.52_±00.03 + 70.39_±00.00	63.31_±00.11
D/gdc	1.868_±0.001	7.55_±00.04 + 70.43_±00.03	73.08_±00.12
D/dmd	1.875_±0.002	3.63_±00.04 + 70.41_±00.00	71.06_±00.11
C/gcc	3.029_±0.000	1.32_±00.00 + 68.80_±00.04	109.29_±00.43
Vala/clang	3.060_±0.000	4.10_±00.03 + 69.79_±00.08	107.55_±00.90
Rust	3.065_±0.000	2.45_±00.05 + 68.57_±00.00	105.15_±00.17
C/clang	3.067_±0.000	1.38_±00.01 + 68.77_±00.02	105.98_±00.38
Zig	3.085_±0.001	1.34_±00.03 + 68.89_±00.00	109.27_±00.08
Nim/gcc	3.088_±0.001	2.43_±00.06 + 67.19_±00.67	111.11_±00.30
Swift	3.094_±0.000	6.49_±00.02 + 68.88_±00.01	108.15_±00.40
Java	3.095_±0.000	38.11_±00.07 + 68.41_±00.13	107.24_±00.38
Vala/gcc	3.125_±0.001	4.13_±00.02 + 69.79_±00.07	114.53_±00.06
Nim/clang	3.136_±0.001	2.88_±00.04 + 70.44_±00.06	107.79_±00.13
V/gcc	3.146_±0.000	1.87_±00.06 + 68.84_±00.00	117.55_±01.78
V/clang	3.147_±0.000	2.22_±00.07 + 68.84_±00.00	115.64_±00.13
Go	3.152_±0.000	3.63_±00.04 + 73.18_±00.09	113.80_±00.18
Go/gccgo	3.159_±0.000	24.91_±00.09 + 73.53_±00.12	112.23_±00.10
Crystal	3.161_±0.001	4.25_±00.02 + 59.67_±00.03	115.88_±00.10
Node.js	3.213_±0.001	41.96_±00.13 + 71.22_±00.19	129.55_±00.09
Python/pypy	3.270_±0.001	65.78_±00.05 + 68.63_±00.01	136.14_±00.12
Scala	3.322_±0.007	65.01_±00.11 + 133.87_±00.15	118.54_±00.41
C#/.NET Core	4.874_±0.001	34.67_±00.10 + 69.34_±00.00	196.06_±00.16
Kotlin/JVM	6.937_±0.001	39.71_±00.10 + 68.16_±00.22	271.50_±00.34
C#/Mono	7.413_±0.001	25.56_±00.12 + 69.54_±00.03	299.15_±00.74
Ruby/truffleruby	14.604_±1.175	351.93_±01.70 + 465.25_±18.48	553.17_±37.67
Ruby/truffleruby (--jvm)	21.393_±0.051	437.31_±33.21 + 961.78_±75.62	804.61_±05.21
Ruby	180.926_±2.288	15.46_±00.03 + 68.58_±00.00	7927.39_±113.83
Ruby (--jit)	182.285_±0.371	272.05_±00.05 + 68.84_±00.00	7995.18_±94.11
Perl	221.133_±0.748	9.56_±00.04 + 599.55_±00.02	9001.25_±37.66
Python	261.473_±0.827	10.58_±00.05 + 68.84_±00.00	10445.40_±55.25
Tcl	298.785_±5.053	7.51_±00.04 + 400.15_±00.03	12660.24_±177.21
Ruby/jruby	404.829_±24.918	266.19_±01.83 + 779.12_±54.18	16982.36_±869.82

Primes

Testing:

generating primes using the optimized sieve of Atkin;
prefix search for their decimal numbers using Trie data structure.

Primes

Language	Time, s	Memory, MiB	Energy, J
Zig	0.059_±0.000	0.75_±00.00 + 49.41_±00.04	2.44_±00.01
Rust	0.093_±0.000	2.19_±00.05 + 77.27_±00.00	3.78_±00.01
Crystal	0.123_±0.000	3.48_±00.02 + 89.43_±00.64	5.54_±00.06
C++/g++	0.132_±0.000	3.54_±00.07 + 85.27_±00.26	5.36_±00.04
C++/clang++	0.143_±0.000	3.08_±00.08 + 75.41_±00.00	5.63_±00.05
V/clang	0.148_±0.000	0.72_±00.00 + 262.80_±00.57	6.15_±00.06
V/gcc	0.150_±0.000	0.71_±00.06 + 259.28_±00.48	6.28_±00.04
Java	0.155_±0.002	36.96_±00.11 + 144.23_±05.61	8.63_±00.15
Node.js	0.256_±0.003	36.96_±00.11 + 181.19_±02.81	12.80_±00.06
Lua/luajit	0.337_±0.001	2.49_±00.03 + 156.75_±00.38	12.87_±00.03
Scala	0.391_±0.005	66.10_±00.30 + 258.71_±02.41	22.50_±00.24
Julia	0.557_±0.001	229.65_±00.18 + 366.99_±02.17	22.00_±00.09
Python/pypy	0.880_±0.002	64.77_±00.07 + 249.36_±00.15	34.55_±00.17
Ruby (--jit)	1.443_±0.005	270.96_±00.02 + 146.80_±00.04	58.79_±00.46
Lua	1.480_±0.005	2.32_±00.01 + 283.97_±00.87	57.87_±00.19
Ruby/truffleruby	1.563_±0.012	287.18_±00.42 + 334.66_±24.08	80.19_±00.54
Ruby/jruby	1.883_±0.026	182.58_±00.29 + 370.42_±55.17	101.51_±01.01
Ruby/truffleruby (--jvm)	1.895_±0.018	348.72_±09.17 + 526.76_±97.23	123.08_±01.27
Ruby	2.032_±0.004	14.36_±00.02 + 146.75_±00.00	82.09_±00.60
Python	4.806_±0.058	10.41_±00.06 + 234.92_±00.64	190.06_±02.19

Tests Execution

Environment

CPU: Intel(R) Xeon(R) E-2324G

Base Docker image: Debian GNU/Linux bookworm/sid

Language	Version
.NET Core	6.0.203
C#/.NET Core	4.1.0-5.22128.4 (5d10d428)
C#/Mono	6.8.0.105
Chez Scheme	9.5.4
Clojure	"1.11.1"
Crystal	1.4.1
D/dmd	v2.100.0
D/gdc	12.1.0
D/ldc2	1.29.0
Elixir	1.12.2
F#/.NET Core	12.0.1.0 for F# 6.0
Go	go1.18.2
Go/gccgo	12.1.0
Haskell	9.2.3
Java	18.0.1.1
Julia	v"1.7.3"
Kotlin	1.6.21
Lua	5.4.4
Lua/luajit	2.1.0-beta3
MLton	20210117
Nim	1.6.6
Node.js	v18.2.0
OCaml	4.14.0
PHP	8.1.5
Perl	v5.34.0
Python	3.10.4
Python/pypy	7.3.9-final0 for Python 3.9.12
Racket	"8.5"
Ruby	3.1.2p20
Ruby/jruby	9.3.4.0
Ruby/truffleruby	22.1.0
Rust	1.61.0
Scala	3.1.2
Swift	5.6.1
Tcl	8.6
V	0.2.4 a3c0a9b
Vala	0.56.1
Zig	0.9.1
clang/clang++	13.0.1
gcc/g++	12.1.0

Using Docker

Build the image:

$ docker build docker/ -t benchmarks

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where <cmd> is:

versions (print installed language versions);
shell (start the shell);
brainfuck bench (build and run Brainfuck bench.b benchmarks);
brainfuck mandel (build and run Brainfuck mandel.b benchmarks);
base64 (build and run Base64 benchmarks);
json (build and run Json benchmarks);
matmul (build and run Matmul benchmarks);
primes (build and run Primes benchmarks);

Please note that the actual measurements provided in the project are taken semi-manually (via shell) as the full update takes days and could have occassional issues in Docker.

There is a ./run.sh that could be used to simplify Docker usage:

./run.sh build (build the image);
./run.sh make versions (run the image with the versions command);
sudo ./run.sh shell (run the image with the `shell' command, sudo is required to read energy levels).

Manual Execution

Makefiles contain recipes for building and executing tests with the proper dependencies. Please use make run (and make run2 where applicable). The measurements are taken using analyze.rb script:

$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]

Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:

$ ATTEMPTS=1 ../analyze.rb make run

Prerequisites

Please use Dockerfile as a reference regarding which packages and tools are required.

For all (optional):

Powercap for reading energy counters in Linux (Debian package powercap-utils).

For Python:

NumPy for matmul tests (Debian package python3-numpy).
UltraJSON for JSON tests (Debian package python3-ujson).

For C++:

Boost for JSON tests (Debian package libboost-dev).
JSON-C for JSON tests (Debian package libjson-c-dev).

For Rust:

libjq for jq test (Debian packages libjq-dev, libonig-dev and environment variable JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/).

For Java, Scala:

Coursier for downloading Maven artifacts.

For Haskell:

network for TCP connectivity between the tests and the test runner.
raw-strings-qq for raw string literals used in tests.

For Perl:

cpanminus for installing modules from CPAN (Debian package cpanminus).

For Vala:

JSON-GLib for JSON tests (Debian package libjson-glib-dev).

Contribution

Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:

The test runner listens on localhost:9001;
All messages are sent using TCP sockets closed immediately after the message has been sent;
There are two messages sent from a test (it establishes the measurement boundary):
1. The beginning message having the format name of the test/tprocess ID (the process ID is used to measure the memory consumption). Please note that the name of the test couldn't use Tab character as it's a delimiter;
2. The end message with any content (mostly it's "stop" for consistency).
The test runner could be unavailable (if the test is launched as is) and the test should gracefully handle it.

Makefile guide

Binary executables

If the test is compiled into a single binary, then two sections of the Makefile require changes:

append a new target (the final binary location) into executables variable;
append the proper target rule.

Compiled artifacts

If the test is compiled, but can't be executed directly as a binary, then three sections of the Makefile require changes:

append a new target (the final artifact location) into artifacts variable;
append the proper target rule to compile the test;
append run[<target_artifact>] rule to run the test.

Scripting language

If the test doesn't require compilation, then two sections of the Makefile requires changes:

append run[<script_file>] into all_runners variable;
append run[<script_file>] rule to run the test.

README update

TOC is regenerated using git-markdown-toc:

./run.sh toc

Docker image update

Debian packages are pinned and updated with the script (first, please ensure that the image is fine with the linter):

./run.sh lint
./run.sh update_apt

Name		Name	Last commit message	Last commit date
Latest commit History 760 Commits
base64		base64
brainfuck		brainfuck
common		common
docker		docker
json		json
matmul		matmul
primes		primes
.gitignore		.gitignore
.rubocop.yml		.rubocop.yml
Directory.Build.props		Directory.Build.props
LICENSE		LICENSE
README.md		README.md
analyze.rb		analyze.rb
repeat.rb		repeat.rb
run.sh		run.sh
xtime.rb		xtime.rb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Table of Content

Overview

Measurements

Test Cases

Brainfuck

bench.b

mandel.b

Base64

Json

Matmul

Primes

Tests Execution

Environment

Using Docker

Manual Execution

Prerequisites

Contribution

Makefile guide

Binary executables

Compiled artifacts

Scripting language

README update

Docker image update

About

Releases

Packages

Languages

License

ColourGrey/benchmarks

Folders and files

Latest commit

History

Repository files navigation

Table of Content

Overview

Measurements

Test Cases

Brainfuck

bench.b

mandel.b

Base64

Json

Matmul

Primes

Tests Execution

Environment

Using Docker

Manual Execution

Prerequisites

Contribution

Makefile guide

Binary executables

Compiled artifacts

Scripting language

README update

Docker image update

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages