Skip to content

Latest commit

 

History

History
393 lines (328 loc) · 15.4 KB

benchmarks.md

File metadata and controls

393 lines (328 loc) · 15.4 KB
Raw

JSON Parse Practical Example

LibraryMean in ms (lower is better) 0.2.70.2.80.3.6
pjson10.183120 ms9.526570 ms9.382459 ms
clj-json41.713927 ms42.148021 ms
boon44.195550 ms45.949513 ms40.701913 ms
cheshire49.784366 ms50.600446 ms
data.json165.176958 ms171.720518 ms 
Goal:  JSON Parse Practical Example
-----
Case:  :data.json
Evaluation count : 360 in 60 samples of 6 calls.
             Execution time mean : 171.720518 ms
    Execution time std-deviation : 3.128919 ms
   Execution time lower quantile : 167.051332 ms ( 2.5%)
   Execution time upper quantile : 176.926498 ms (97.5%)
                   Overhead used : 1.881626 ns

Found 1 outliers in 60 samples (1.6667 %)
        low-severe       1 (1.6667 %)
 Variance from outliers : 7.7886 % Variance is slightly inflated by outliers

Case:  :pjson
Evaluation count : 6360 in 60 samples of 106 calls.
             Execution time mean : 9.382459 ms
    Execution time std-deviation : 96.696634 µs
   Execution time lower quantile : 9.258054 ms ( 2.5%)
   Execution time upper quantile : 9.694433 ms (97.5%)
                   Overhead used : 1.553512 ns

Found 4 outliers in 60 samples (6.6667 %)
        low-severe       1 (1.6667 %)
        low-mild         3 (5.0000 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

Case:  :cheshire
Evaluation count : 1200 in 60 samples of 20 calls.
             Execution time mean : 50.600446 ms
    Execution time std-deviation : 1.236219 ms
   Execution time lower quantile : 49.122398 ms ( 2.5%)
   Execution time upper quantile : 54.349364 ms (97.5%)
                   Overhead used : 1.881626 ns

Found 3 outliers in 60 samples (5.0000 %)
        low-severe       2 (3.3333 %)
        low-mild         1 (1.6667 %)
 Variance from outliers : 12.5577 % Variance is moderately inflated by outliers

Case:  :boon
Evaluation count : 1500 in 60 samples of 25 calls.
             Execution time mean : 40.701913 ms
    Execution time std-deviation : 587.048788 µs
   Execution time lower quantile : 39.874727 ms ( 2.5%)
   Execution time upper quantile : 42.248703 ms (97.5%)
                   Overhead used : 1.553512 ns

Found 3 outliers in 60 samples (5.0000 %)
        low-severe       3 (5.0000 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

Case:  :clj-json
Evaluation count : 1500 in 60 samples of 25 calls.
             Execution time mean : 42.148021 ms
    Execution time std-deviation : 2.859100 ms
   Execution time lower quantile : 40.430278 ms ( 2.5%)
   Execution time upper quantile : 49.266426 ms (97.5%)
                   Overhead used : 1.881626 ns

Found 5 outliers in 60 samples (8.3333 %)
        low-severe       3 (5.0000 %)
        low-mild         2 (3.3333 %)
 Variance from outliers : 51.7236 % Variance is severely inflated by outliers

JSON Parse Benchmark

LibraryMean in ms (lower is better) 0.2.70.2.80.3.6
pjson3.471202 ms3.397253 ms3.239881 ms
clj-json30.000884 ms30.696739 ms
boon7.775955 ms8.178839 ms6.669186 ms
cheshire26.599338 ms27.984381 ms
data.json74.359619 ms76.775012 ms 
Goal:  JSON Parse Benchmark
-----
Case:  :clj-json
Evaluation count : 1980 in 60 samples of 33 calls.
             Execution time mean : 30.696739 ms
    Execution time std-deviation : 546.312660 µs
   Execution time lower quantile : 29.917998 ms ( 2.5%)
   Execution time upper quantile : 32.038518 ms (97.5%)
                   Overhead used : 1.881626 ns

Found 4 outliers in 60 samples (6.6667 %)
        low-severe       4 (6.6667 %)
 Variance from outliers : 7.7624 % Variance is slightly inflated by outliers

Case:  :boon
Evaluation count : 9000 in 60 samples of 150 calls.
             Execution time mean : 6.669186 ms
    Execution time std-deviation : 146.054908 µs
   Execution time lower quantile : 6.563548 ms ( 2.5%)
   Execution time upper quantile : 7.100086 ms (97.5%)
                   Overhead used : 1.553512 ns

Found 5 outliers in 60 samples (8.3333 %)
        low-severe       1 (1.6667 %)
        low-mild         4 (6.6667 %)
 Variance from outliers : 9.4570 % Variance is slightly inflated by outliers

Case:  :pjson
Evaluation count : 18780 in 60 samples of 313 calls.
             Execution time mean : 3.239881 ms
    Execution time std-deviation : 35.395760 µs
   Execution time lower quantile : 3.206980 ms ( 2.5%)
   Execution time upper quantile : 3.329477 ms (97.5%)
                   Overhead used : 1.553512 ns

Found 5 outliers in 60 samples (8.3333 %)
        low-severe       3 (5.0000 %)
        low-mild         2 (3.3333 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

Case:  :cheshire
Evaluation count : 2160 in 60 samples of 36 calls.
             Execution time mean : 27.984381 ms
    Execution time std-deviation : 695.037351 µs
   Execution time lower quantile : 27.106720 ms ( 2.5%)
   Execution time upper quantile : 29.370169 ms (97.5%)
                   Overhead used : 1.881626 ns

Found 5 outliers in 60 samples (8.3333 %)
        low-severe       4 (6.6667 %)
        low-mild         1 (1.6667 %)
 Variance from outliers : 12.5828 % Variance is moderately inflated by outliers

Case:  :data.json
Evaluation count : 840 in 60 samples of 14 calls.
             Execution time mean : 76.775012 ms
    Execution time std-deviation : 1.458656 ms
   Execution time lower quantile : 75.032641 ms ( 2.5%)
   Execution time upper quantile : 80.653407 ms (97.5%)
                   Overhead used : 1.881626 ns

Found 5 outliers in 60 samples (8.3333 %)
        low-severe       3 (5.0000 %)
        low-mild         2 (3.3333 %)
 Variance from outliers : 7.8323 % Variance is slightly inflated by outliers

JSON Parse Benchmark - parse all fields and substructures

LibraryMean in ms (lower is better) 0.2.70.2.80.3.6
pjson210.109698 ms214.128818 ms154.815778 ms
clj-json234.800428 ms244.911982 ms185.095822 ms
boon226.109138 ms235.744988 ms167.414613 ms
cheshire233.513268 ms242.169108 ms180.337285 ms
data.json282.268952 ms292.574602 ms221.465102 ms 
Goal:  JSON Parse Benchmark - parse all fields and substructures
-----
Case:  :data.json
Evaluation count : 300 in 60 samples of 5 calls.
             Execution time mean : 221.465102 ms
    Execution time std-deviation : 2.685839 ms
   Execution time lower quantile : 217.661820 ms ( 2.5%)
   Execution time upper quantile : 227.296331 ms (97.5%)
                   Overhead used : 1.553512 ns

Found 4 outliers in 60 samples (6.6667 %)
        low-severe       3 (5.0000 %)
        low-mild         1 (1.6667 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

Case:  :boon
Evaluation count : 420 in 60 samples of 7 calls.
             Execution time mean : 167.414613 ms
    Execution time std-deviation : 1.828060 ms
   Execution time lower quantile : 165.567706 ms ( 2.5%)
   Execution time upper quantile : 172.131859 ms (97.5%)
                   Overhead used : 1.553512 ns

Found 4 outliers in 60 samples (6.6667 %)
        low-severe       3 (5.0000 %)
        low-mild         1 (1.6667 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

Case:  :cheshire
Evaluation count : 360 in 60 samples of 6 calls.
             Execution time mean : 180.337285 ms
    Execution time std-deviation : 3.151541 ms
   Execution time lower quantile : 176.927905 ms ( 2.5%)
   Execution time upper quantile : 187.636870 ms (97.5%)
                   Overhead used : 1.553512 ns

Found 8 outliers in 60 samples (13.3333 %)
        low-severe       7 (11.6667 %)
        low-mild         1 (1.6667 %)
 Variance from outliers : 6.2877 % Variance is slightly inflated by outliers

Case:  :pjson
Evaluation count : 420 in 60 samples of 7 calls.
             Execution time mean : 154.815778 ms
    Execution time std-deviation : 1.529529 ms
   Execution time lower quantile : 153.082568 ms ( 2.5%)
   Execution time upper quantile : 157.792013 ms (97.5%)
                   Overhead used : 1.553512 ns

Found 3 outliers in 60 samples (5.0000 %)
        low-severe       2 (3.3333 %)
        low-mild         1 (1.6667 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

Case:  :clj-json
Evaluation count : 360 in 60 samples of 6 calls.
             Execution time mean : 185.095822 ms
    Execution time std-deviation : 2.106442 ms
   Execution time lower quantile : 182.398961 ms ( 2.5%)
   Execution time upper quantile : 190.273053 ms (97.5%)
                   Overhead used : 1.553512 ns

Found 2 outliers in 60 samples (3.3333 %)
        low-severe       2 (3.3333 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

JSON Encode Map

This benchmarks aims to test encoding speed from standard clojure/java data structures.
The test message is parsed with clj-json which does this without any lazy intermediates.

LibraryMean in ms (lower is better) 0.2.70.2.80.3.6
pjson9.881341 ms10.291857 ms9.577133 ms
clj-json9.240113 ms8.785915 ms7.440156 ms
boon60.509740 ms65.338146 ms56.358606 ms
cheshire19.348805 ms20.054495 ms21.417734 ms
data.json76.980829 ms79.525906 ms73.388868 ms 
Goal:  JSON Encode Map Benchmark
-----
Case:  :boon
Evaluation count : 1080 in 60 samples of 18 calls.
             Execution time mean : 56.358606 ms
    Execution time std-deviation : 1.507475 ms
   Execution time lower quantile : 55.202649 ms ( 2.5%)
   Execution time upper quantile : 59.223079 ms (97.5%)
                   Overhead used : 1.553512 ns

Found 4 outliers in 60 samples (6.6667 %)
        low-severe       1 (1.6667 %)
        low-mild         3 (5.0000 %)
 Variance from outliers : 14.1810 % Variance is moderately inflated by outliers

Case:  :pjson
Evaluation count : 6300 in 60 samples of 105 calls.
             Execution time mean : 9.577133 ms
    Execution time std-deviation : 65.927621 µs
   Execution time lower quantile : 9.473455 ms ( 2.5%)
   Execution time upper quantile : 9.736950 ms (97.5%)
                   Overhead used : 1.553512 ns

Found 3 outliers in 60 samples (5.0000 %)
        low-severe       1 (1.6667 %)
        low-mild         2 (3.3333 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

Case:  :data.json
Evaluation count : 840 in 60 samples of 14 calls.
             Execution time mean : 73.388868 ms
    Execution time std-deviation : 416.120409 µs
   Execution time lower quantile : 72.807563 ms ( 2.5%)
   Execution time upper quantile : 74.423537 ms (97.5%)
                   Overhead used : 1.553512 ns

Found 2 outliers in 60 samples (3.3333 %)
        low-severe       2 (3.3333 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

Case:  :cheshire
Evaluation count : 2820 in 60 samples of 47 calls.
             Execution time mean : 21.417734 ms
    Execution time std-deviation : 186.582387 µs
   Execution time lower quantile : 21.146894 ms ( 2.5%)
   Execution time upper quantile : 21.862683 ms (97.5%)
                   Overhead used : 1.553512 ns

Found 3 outliers in 60 samples (5.0000 %)
        low-severe       3 (5.0000 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

Case:  :clj-json
Evaluation count : 8160 in 60 samples of 136 calls.
             Execution time mean : 7.440156 ms
    Execution time std-deviation : 44.436823 µs
   Execution time lower quantile : 7.360257 ms ( 2.5%)
   Execution time upper quantile : 7.527837 ms (97.5%)
                   Overhead used : 1.553512 ns

Found 2 outliers in 60 samples (3.3333 %)
        low-severe       2 (3.3333 %)
 Variance from outliers : 1.6389 % Variance is slightly inflated by outliers

JSON Encode Simple

This benchmarks reads the message using each API's decode/parse functions and then without edits
encodes the message back to json.

Note: pjson can remember the original json message for sub messages, such that if no edits were made the messages' original json string is written back.

Summary of the benchmark results are below (in order of faster to slowest).

LibraryMean in ms (lower is better) 0.2.70.2.80.3.6
pjson2.380149 ms2.394470 ms2.431828 ms
clj-json9.208672 ms9.950834 ms8.026480 ms
boon10.196466 ms11.121667 ms9.488874 ms
cheshire19.375317 ms20.302390 ms21.962842 ms
data.json74.689144 ms82.368058 ms78.556321 ms 
Goal:  JSON Encode Simple
-----
Case:  :clj-json
Evaluation count : 7800 in 60 samples of 130 calls.
             Execution time mean : 8.026480 ms
    Execution time std-deviation : 305.671832 µs
   Execution time lower quantile : 7.589653 ms ( 2.5%)
   Execution time upper quantile : 8.600770 ms (97.5%)
                   Overhead used : 1.553512 ns

Case:  :pjson
Evaluation count : 25080 in 60 samples of 418 calls.
             Execution time mean : 2.431828 ms
    Execution time std-deviation : 42.347751 µs
   Execution time lower quantile : 2.352209 ms ( 2.5%)
   Execution time upper quantile : 2.506976 ms (97.5%)
                   Overhead used : 1.553512 ns

Case:  :data.json
Evaluation count : 840 in 60 samples of 14 calls.
             Execution time mean : 78.556321 ms
    Execution time std-deviation : 2.979267 ms
   Execution time lower quantile : 73.135290 ms ( 2.5%)
   Execution time upper quantile : 83.312295 ms (97.5%)
                   Overhead used : 1.553512 ns

Case:  :cheshire
Evaluation count : 2880 in 60 samples of 48 calls.
             Execution time mean : 21.962842 ms
    Execution time std-deviation : 756.145386 µs
   Execution time lower quantile : 21.024825 ms ( 2.5%)
   Execution time upper quantile : 23.578767 ms (97.5%)
                   Overhead used : 1.553512 ns

Case:  :boon
Evaluation count : 6360 in 60 samples of 106 calls.
             Execution time mean : 9.488874 ms
    Execution time std-deviation : 419.549008 µs
   Execution time lower quantile : 8.984589 ms ( 2.5%)
   Execution time upper quantile : 10.278705 ms (97.5%)
                   Overhead used : 1.553512 ns

Number parse Benchmark

This benchmark is used to show the difference between the manual string to integer reading in pjson and the java Long.valueOf

LibraryMean in ms (lower is better) 0.2.8
pjson number-util-parse512.869024 ns
java Long.valueOf38.067585 ns