Skip to content

Commit

Permalink
Clean up benchmark .spacetime in CI (#1968)
Browse files Browse the repository at this point in the history
  • Loading branch information
@kazimuth
kazimuth authored Nov 8, 2024
1 parent e9e5a97 commit dd73f76
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions .github/workflows/benchmarks.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@ jobs:
echo "Running benchmarks with sqlite"
fi
pushd crates/bench
rm -rf .spacetime
cargo bench --bench generic -- --save-baseline "$BASELINE_NAME" "$BENCH_FILTER"
# sticker price benchmark
cargo bench --bench generic -- --save-baseline "$BASELINE_NAME" 'stdb_module/disk/update_bulk'
Expand Down Expand Up @@ -287,6 +288,7 @@ jobs:
echo "Running master callgrind benchmarks"
fi
pushd crates/bench
rm -rf .spacetime
cargo bench --bench callgrind -- --save-summary pretty-json
cargo run --bin summarize pack-callgrind "$BASELINE_NAME"
popd
Expand Down

2 comments on commit dd73f76

@github-actions
Copy link

@github-actions github-actions bot commented on dd73f76 Nov 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Criterion benchmark results

Criterion benchmark report

YOU SHOULD PROBABLY IGNORE THESE RESULTS.

Criterion is a wall time based benchmarking system that is extremely noisy when run on CI. We collect these results for longitudinal analysis, but they are not reliable for comparing individual PRs.

Go look at the callgrind report instead.

empty

db on disk new latency old latency new throughput old throughput
sqlite 💿 419.8±2.00ns 421.2±4.72ns - -
sqlite 🧠 413.5±2.51ns 411.6±2.68ns - -
stdb_raw 💿 775.6±1.29ns 774.9±1.22ns - -
stdb_raw 🧠 776.2±1.53ns 774.2±1.02ns - -

insert_1

db on disk schema indices preload new latency old latency new throughput old throughput

insert_bulk

db on disk schema indices preload count new latency old latency new throughput old throughput
sqlite 💿 u32_u64_str btree_each_column 2048 256 591.0±0.70µs 586.9±0.86µs 1692 tx/sec 1703 tx/sec
sqlite 💿 u32_u64_str unique_0 2048 256 153.9±1.04µs 152.7±0.51µs 6.3 Ktx/sec 6.4 Ktx/sec
sqlite 💿 u32_u64_u64 btree_each_column 2048 256 472.3±0.43µs 472.4±0.65µs 2.1 Ktx/sec 2.1 Ktx/sec
sqlite 💿 u32_u64_u64 unique_0 2048 256 140.9±0.88µs 137.7±0.65µs 6.9 Ktx/sec 7.1 Ktx/sec
sqlite 🧠 u32_u64_str btree_each_column 2048 256 450.8±0.54µs 447.9±0.87µs 2.2 Ktx/sec 2.2 Ktx/sec
sqlite 🧠 u32_u64_str unique_0 2048 256 125.6±0.41µs 123.6±0.58µs 7.8 Ktx/sec 7.9 Ktx/sec
sqlite 🧠 u32_u64_u64 btree_each_column 2048 256 372.8±0.66µs 371.0±0.65µs 2.6 Ktx/sec 2.6 Ktx/sec
sqlite 🧠 u32_u64_u64 unique_0 2048 256 107.8±0.65µs 109.3±1.05µs 9.1 Ktx/sec 8.9 Ktx/sec
stdb_raw 💿 u32_u64_str btree_each_column 2048 256 604.6±26.90µs 596.7±23.27µs 1653 tx/sec 1675 tx/sec
stdb_raw 💿 u32_u64_str unique_0 2048 256 408.7±53.76µs 483.7±35.95µs 2.4 Ktx/sec 2.0 Ktx/sec
stdb_raw 💿 u32_u64_u64 btree_each_column 2048 256 368.9±10.25µs 320.2±10.39µs 2.6 Ktx/sec 3.0 Ktx/sec
stdb_raw 💿 u32_u64_u64 unique_0 2048 256 344.6±6.96µs 331.4±19.71µs 2.8 Ktx/sec 2.9 Ktx/sec
stdb_raw 🧠 u32_u64_str btree_each_column 2048 256 297.0±0.38µs 296.2±0.16µs 3.3 Ktx/sec 3.3 Ktx/sec
stdb_raw 🧠 u32_u64_str unique_0 2048 256 229.4±0.31µs 227.5±0.44µs 4.3 Ktx/sec 4.3 Ktx/sec
stdb_raw 🧠 u32_u64_u64 btree_each_column 2048 256 235.9±0.14µs 234.6±0.12µs 4.1 Ktx/sec 4.2 Ktx/sec
stdb_raw 🧠 u32_u64_u64 unique_0 2048 256 207.1±0.23µs 207.1±0.29µs 4.7 Ktx/sec 4.7 Ktx/sec

iterate

db on disk schema indices new latency old latency new throughput old throughput
sqlite 💿 u32_u64_str unique_0 24.7±0.15µs 23.9±0.02µs 39.6 Ktx/sec 40.8 Ktx/sec
sqlite 💿 u32_u64_u64 unique_0 22.3±0.08µs 21.0±0.28µs 43.9 Ktx/sec 46.5 Ktx/sec
sqlite 🧠 u32_u64_str unique_0 21.9±0.09µs 21.4±0.08µs 44.5 Ktx/sec 45.6 Ktx/sec
sqlite 🧠 u32_u64_u64 unique_0 19.7±0.12µs 18.3±0.03µs 49.6 Ktx/sec 53.3 Ktx/sec
stdb_raw 💿 u32_u64_str unique_0 4.9±0.00µs 4.9±0.00µs 199.8 Ktx/sec 200.4 Ktx/sec
stdb_raw 💿 u32_u64_u64 unique_0 4.8±0.00µs 4.8±0.00µs 203.8 Ktx/sec 204.4 Ktx/sec
stdb_raw 🧠 u32_u64_str unique_0 4.9±0.00µs 4.9±0.00µs 200.0 Ktx/sec 200.3 Ktx/sec
stdb_raw 🧠 u32_u64_u64 unique_0 4.8±0.00µs 4.8±0.00µs 203.9 Ktx/sec 204.5 Ktx/sec

find_unique

db on disk key type preload new latency old latency new throughput old throughput

filter

db on disk key type index strategy load count new latency old latency new throughput old throughput
sqlite 💿 string index 2048 256 69.9±0.46µs 67.6±0.11µs 14.0 Ktx/sec 14.5 Ktx/sec
sqlite 💿 u64 index 2048 256 65.1±0.19µs 63.1±0.27µs 15.0 Ktx/sec 15.5 Ktx/sec
sqlite 🧠 string index 2048 256 65.2±0.28µs 64.0±0.19µs 15.0 Ktx/sec 15.3 Ktx/sec
sqlite 🧠 u64 index 2048 256 59.1±0.55µs 57.4±0.11µs 16.5 Ktx/sec 17.0 Ktx/sec
stdb_raw 💿 string index 2048 256 5.1±0.00µs 5.2±0.01µs 192.7 Ktx/sec 188.7 Ktx/sec
stdb_raw 💿 u64 index 2048 256 5.0±0.00µs 5.1±0.00µs 195.8 Ktx/sec 192.8 Ktx/sec
stdb_raw 🧠 string index 2048 256 5.1±0.00µs 5.2±0.00µs 192.8 Ktx/sec 189.1 Ktx/sec
stdb_raw 🧠 u64 index 2048 256 5.0±0.00µs 5.1±0.00µs 195.9 Ktx/sec 192.7 Ktx/sec

serialize

schema format count new latency old latency new throughput old throughput
u32_u64_str bflatn_to_bsatn_fast_path 100 3.6±0.00µs 3.6±0.00µs 26.5 Mtx/sec 26.5 Mtx/sec
u32_u64_str bflatn_to_bsatn_slow_path 100 3.6±0.02µs 3.6±0.02µs 26.5 Mtx/sec 26.6 Mtx/sec
u32_u64_str bsatn 100 15.7±0.12ns 15.5±0.10ns 5.9 Gtx/sec 6.0 Gtx/sec
u32_u64_str bsatn 100 2.4±0.00µs 2.4±0.01µs 39.6 Mtx/sec 39.5 Mtx/sec
u32_u64_str json 100 5.1±0.04µs 5.1±0.04µs 18.9 Mtx/sec 18.8 Mtx/sec
u32_u64_str json 100 8.2±0.06µs 8.2±0.05µs 11.6 Mtx/sec 11.7 Mtx/sec
u32_u64_str product_value 100 1018.5±3.21ns 1018.6±0.96ns 93.6 Mtx/sec 93.6 Mtx/sec
u32_u64_u64 bflatn_to_bsatn_fast_path 100 1018.8±3.86ns 1006.1±11.65ns 93.6 Mtx/sec 94.8 Mtx/sec
u32_u64_u64 bflatn_to_bsatn_slow_path 100 2.8±0.03µs 2.8±0.03µs 34.2 Mtx/sec 34.0 Mtx/sec
u32_u64_u64 bsatn 100 14.8±0.01ns 14.8±0.04ns 6.3 Gtx/sec 6.3 Gtx/sec
u32_u64_u64 bsatn 100 1741.0±21.45ns 1734.0±21.74ns 54.8 Mtx/sec 55.0 Mtx/sec
u32_u64_u64 json 100 3.1±0.01µs 3.1±0.04µs 30.7 Mtx/sec 30.4 Mtx/sec
u32_u64_u64 json 100 6.0±0.02µs 5.9±0.03µs 15.9 Mtx/sec 16.1 Mtx/sec
u32_u64_u64 product_value 100 1016.0±3.81ns 1015.7±0.77ns 93.9 Mtx/sec 93.9 Mtx/sec
u64_u64_u32 bflatn_to_bsatn_fast_path 100 767.4±9.09ns 758.3±5.27ns 124.3 Mtx/sec 125.8 Mtx/sec
u64_u64_u32 bflatn_to_bsatn_slow_path 100 2.8±0.00µs 2.8±0.01µs 34.2 Mtx/sec 34.1 Mtx/sec
u64_u64_u32 bsatn 100 1745.2±20.38ns 1740.4±23.82ns 54.6 Mtx/sec 54.8 Mtx/sec
u64_u64_u32 bsatn 100 706.0±0.63ns 705.4±0.84ns 135.1 Mtx/sec 135.2 Mtx/sec
u64_u64_u32 json 100 3.1±0.01µs 3.2±0.04µs 30.3 Mtx/sec 30.0 Mtx/sec
u64_u64_u32 json 100 5.8±0.01µs 5.9±0.04µs 16.3 Mtx/sec 16.3 Mtx/sec
u64_u64_u32 product_value 100 1014.1±0.57ns 1014.1±0.59ns 94.0 Mtx/sec 94.0 Mtx/sec

stdb_module_large_arguments

arg size new latency old latency new throughput old throughput
64KiB 97.4±7.28µs 104.6±6.28µs - -

stdb_module_print_bulk

line count new latency old latency new throughput old throughput
1 53.9±4.08µs 56.5±5.19µs - -
100 603.9±6.34µs 602.3±4.61µs - -
1000 4.0±0.84ms 5.4±0.03ms - -

remaining

name new latency old latency new throughput old throughput
special/db_game/circles/load=10 49.8±8.65ms 53.0±1.29ms - -
special/db_game/circles/load=100 37.3±1.65ms 38.5±2.71ms - -
special/db_game/ia_loop/load=500 149.5±1.07ms 150.6±1.07ms - -
special/db_game/ia_loop/load=5000 5.4±0.02s 5.3±0.03s - -
sqlite/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 55.2±0.28µs 53.7±0.28µs 17.7 Ktx/sec 18.2 Ktx/sec
sqlite/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 48.2±0.14µs 46.1±0.11µs 20.3 Ktx/sec 21.2 Ktx/sec
sqlite/🧠/update_bulk/u32_u64_str/unique_0/load=2048/count=256 40.7±0.34µs 39.1±0.24µs 24.0 Ktx/sec 25.0 Ktx/sec
sqlite/🧠/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 36.6±0.17µs 34.9±0.18µs 26.7 Ktx/sec 28.0 Ktx/sec
stdb_module/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 1262.7±8.08µs 1268.4±12.89µs 791 tx/sec 788 tx/sec
stdb_module/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 1009.2±9.15µs 1009.6±7.88µs 990 tx/sec 990 tx/sec
stdb_raw/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 534.1±7.03µs 622.0±16.94µs 1872 tx/sec 1607 tx/sec
stdb_raw/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 478.7±7.29µs 415.2±11.69µs 2.0 Ktx/sec 2.4 Ktx/sec
stdb_raw/🧠/update_bulk/u32_u64_str/unique_0/load=2048/count=256 365.6±0.26µs 360.6±0.20µs 2.7 Ktx/sec 2.7 Ktx/sec
stdb_raw/🧠/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 327.0±0.48µs 326.7±0.20µs 3.0 Ktx/sec 3.0 Ktx/sec

@github-actions
Copy link

@github-actions github-actions bot commented on dd73f76 Nov 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Callgrind benchmark results

Callgrind Benchmark Report

These benchmarks were run using callgrind,
an instruction-level profiler. They allow comparisons between sqlite (sqlite), SpacetimeDB running through a module (stdb_module), and the underlying SpacetimeDB data storage engine (stdb_raw). Callgrind emulates a CPU to collect the below estimates.

Measurement changes larger than five percent are in bold.

In-memory benchmarks

callgrind: empty transaction

db total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw 6399 6399 0.00% 6523 6523 0.00%
sqlite 5579 5579 0.00% 6019 6019 0.00%

callgrind: filter

db schema indices count preload _column data_type total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str no_index 64 128 1 u64 76593 76593 0.00% 77013 76981 0.04%
stdb_raw u32_u64_str no_index 64 128 2 string 120180 119091 0.91% 120792 119779 0.85%
stdb_raw u32_u64_str btree_each_column 64 128 2 string 25085 25083 0.01% 25641 25627 0.05%
stdb_raw u32_u64_str btree_each_column 64 128 1 u64 24051 24051 0.00% 24419 24451 -0.13%
sqlite u32_u64_str no_index 64 128 2 string 144695 144695 0.00% 146257 146261 -0.00%
sqlite u32_u64_str no_index 64 128 1 u64 124044 124044 0.00% 125334 125338 -0.00%
sqlite u32_u64_str btree_each_column 64 128 1 u64 131361 131361 0.00% 132811 132811 0.00%
sqlite u32_u64_str btree_each_column 64 128 2 string 134494 134494 0.00% 136096 136100 -0.00%

callgrind: insert bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 64 128 877044 875757 0.15% 931998 899593 3.60%
stdb_raw u32_u64_str btree_each_column 64 128 1026659 1023785 0.28% 1068451 1064997 0.32%
sqlite u32_u64_str unique_0 64 128 398320 398320 0.00% 415178 415178 0.00%
sqlite u32_u64_str btree_each_column 64 128 983637 983637 0.00% 1020521 1020529 -0.00%

callgrind: iterate

db schema indices count total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 153726 153726 0.00% 153866 153866 0.00%
stdb_raw u32_u64_str unique_0 64 16751 16751 0.00% 16875 16875 0.00%
sqlite u32_u64_str unique_0 1024 1067255 1067273 -0.00% 1070663 1070681 -0.00%
sqlite u32_u64_str unique_0 64 76207 76207 0.00% 77445 77445 0.00%

callgrind: serialize_product_value

count format total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
64 json 47528 47528 0.00% 50282 50282 0.00%
64 bsatn 25509 25509 0.00% 27821 27821 0.00%
16 bsatn 8200 8200 0.00% 9628 9628 0.00%
16 json 12188 12188 0.00% 14194 14194 0.00%

callgrind: update bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 1024 20076320 20084936 -0.04% 20646444 20662240 -0.08%
stdb_raw u32_u64_str unique_0 64 128 1285051 1284386 0.05% 1327185 1359554 -2.38%
sqlite u32_u64_str unique_0 1024 1024 1802182 1802182 0.00% 1811362 1811362 0.00%
sqlite u32_u64_str unique_0 64 128 128528 128528 0.00% 131492 131492 0.00%
On-disk benchmarks

callgrind: empty transaction

db total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw 6404 6404 0.00% 6544 6544 0.00%
sqlite 5621 5621 0.00% 6121 6121 0.00%

callgrind: filter

db schema indices count preload _column data_type total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str no_index 64 128 1 u64 76598 76598 0.00% 76982 76982 0.00%
stdb_raw u32_u64_str no_index 64 128 2 string 119096 119096 0.00% 119852 119820 0.03%
stdb_raw u32_u64_str btree_each_column 64 128 2 string 25333 25088 0.98% 25857 25572 1.11%
stdb_raw u32_u64_str btree_each_column 64 128 1 u64 24056 24056 0.00% 24400 24400 0.00%
sqlite u32_u64_str no_index 64 128 1 u64 125965 125965 0.00% 127547 127547 0.00%
sqlite u32_u64_str no_index 64 128 2 string 146616 146616 0.00% 148450 148446 0.00%
sqlite u32_u64_str btree_each_column 64 128 2 string 136616 136616 0.00% 138724 138724 0.00%
sqlite u32_u64_str btree_each_column 64 128 1 u64 133457 133457 0.00% 135341 135341 0.00%

callgrind: insert bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 64 128 826664 827011 -0.04% 880466 880749 -0.03%
stdb_raw u32_u64_str btree_each_column 64 128 975398 974099 0.13% 1046160 1044871 0.12%
sqlite u32_u64_str unique_0 64 128 415857 415857 0.00% 431871 431875 -0.00%
sqlite u32_u64_str btree_each_column 64 128 1021898 1021898 0.00% 1057614 1057614 0.00%

callgrind: iterate

db schema indices count total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 153731 153731 0.00% 153855 153855 0.00%
stdb_raw u32_u64_str unique_0 64 16756 16756 0.00% 16876 16880 -0.02%
sqlite u32_u64_str unique_0 1024 1070323 1070323 0.00% 1074097 1074097 0.00%
sqlite u32_u64_str unique_0 64 77973 77973 0.00% 79303 79303 0.00%

callgrind: serialize_product_value

count format total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
64 json 47528 47528 0.00% 50282 50282 0.00%
64 bsatn 25509 25509 0.00% 27821 27821 0.00%
16 bsatn 8200 8200 0.00% 9628 9628 0.00%
16 json 12188 12188 0.00% 14194 14194 0.00%

callgrind: update bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 1024 19002051 19002577 -0.00% 19656511 19644985 0.06%
stdb_raw u32_u64_str unique_0 64 128 1237908 1237396 0.04% 1309022 1309094 -0.01%
sqlite u32_u64_str unique_0 1024 1024 1809743 1809761 -0.00% 1818455 1818485 -0.00%
sqlite u32_u64_str unique_0 64 128 132654 132654 0.00% 135834 135834 0.00%

Please sign in to comment.