Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[do not review, testing benchmarks] #2053

Closed
wants to merge 3 commits into from
Closed

[do not review, testing benchmarks] #2053

wants to merge 3 commits into from

Conversation

RReverser
Copy link
Member

Description of Changes

Please describe your change, mention any related tickets, and so on here.

API and ABI breaking changes

If this is an API or ABI breaking change, please apply the
corresponding GitHub label.

Expected complexity level and risk

How complicated do you think these changes are? Grade on a scale from 1 to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex change.

This complexity rating applies not only to the complexity apparent in the diff,
but also to its interactions with existing and future code.

If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning ways.

Testing

Describe any testing you've done, and any testing you'd like your reviewers to do,
so that you're confident that all the changes work as expected!

  • Write a test you've completed here.
  • Write a test you want a reviewer to do here, so they can check it off when they're satisfied.

@RReverser
Copy link
Member Author

benchmarks please

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 11, 2024 •
edited
Loading

Benchmarking failed. Please check the workflow run for details.

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 11, 2024 •
edited
Loading

Callgrind benchmark results

Callgrind Benchmark Report

These benchmarks were run using callgrind,
an instruction-level profiler. They allow comparisons between sqlite (sqlite), SpacetimeDB running through a module (stdb_module), and the underlying SpacetimeDB data storage engine (stdb_raw). Callgrind emulates a CPU to collect the below estimates.

Measurement changes larger than five percent are in bold.

In-memory benchmarks

callgrind: empty transaction

db total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw 6455 6455 0.00% 6559 6555 0.06%
sqlite 5609 5609 0.00% 5951 5951 0.00%

callgrind: filter

db schema indices count preload _column data_type total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str no_index 64 128 1 u64 75267 75267 0.00% 75711 75711 0.00%
stdb_raw u32_u64_str no_index 64 128 2 string 117551 117551 0.00% 118389 118321 0.06%
stdb_raw u32_u64_str btree_each_column 64 128 2 string 25087 25072 0.06% 25633 25634 -0.00%
stdb_raw u32_u64_str btree_each_column 64 128 1 u64 24039 24039 0.00% 24471 24471 0.00%
sqlite u32_u64_str no_index 64 128 2 string 144351 144351 0.00% 145901 145901 0.00%
sqlite u32_u64_str no_index 64 128 1 u64 123742 123742 0.00% 125194 125194 0.00%
sqlite u32_u64_str btree_each_column 64 128 1 u64 131059 131059 0.00% 132683 132687 -0.00%
sqlite u32_u64_str btree_each_column 64 128 2 string 134158 134158 0.00% 135816 135824 -0.01%

callgrind: insert bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 64 128 815301 815455 -0.02% 832745 832871 -0.02%
stdb_raw u32_u64_str btree_each_column 64 128 1040885 1040515 0.04% 1069915 1100193 -2.75%
sqlite u32_u64_str unique_0 64 128 399360 399360 0.00% 415582 415582 0.00%
sqlite u32_u64_str btree_each_column 64 128 984611 984611 0.00% 1024099 1024099 0.00%

callgrind: iterate

db schema indices count total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 138448 138448 0.00% 138568 138564 0.00%
stdb_raw u32_u64_str unique_0 64 15873 15873 0.00% 15981 15981 0.00%
sqlite u32_u64_str unique_0 1024 1042718 1042724 -0.00% 1046104 1046110 -0.00%
sqlite u32_u64_str unique_0 64 74704 74704 0.00% 75892 75892 0.00%

callgrind: serialize_product_value

count format total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
64 json 47528 47528 0.00% 50316 50316 0.00%
64 bsatn 25509 25509 0.00% 27787 27787 0.00%
16 bsatn 8200 8200 0.00% 9628 9628 0.00%
16 json 12188 12188 0.00% 14194 14194 0.00%

callgrind: update bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 1024 19050374 19061720 -0.06% 19532324 19597048 -0.33%
stdb_raw u32_u64_str unique_0 64 128 1237166 1251271 -1.13% 1273812 1318901 -3.42%
sqlite u32_u64_str unique_0 1024 1024 1802137 1802155 -0.00% 1811113 1811143 -0.00%
sqlite u32_u64_str unique_0 64 128 128540 128540 0.00% 131306 131306 0.00%
On-disk benchmarks

callgrind: empty transaction

db total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw 6460 6460 0.00% 6564 6564 0.00%
sqlite 5651 5651 0.00% 6019 6019 0.00%

callgrind: filter

db schema indices count preload _column data_type total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str no_index 64 128 1 u64 75272 75272 0.00% 75704 75716 -0.02%
stdb_raw u32_u64_str no_index 64 128 2 string 117556 117548 0.01% 118190 118246 -0.05%
stdb_raw u32_u64_str btree_each_column 64 128 2 string 25093 25111 -0.07% 25619 25657 -0.15%
stdb_raw u32_u64_str btree_each_column 64 128 1 u64 24044 24044 0.00% 24460 24464 -0.02%
sqlite u32_u64_str no_index 64 128 1 u64 125663 125663 0.00% 127311 127311 0.00%
sqlite u32_u64_str no_index 64 128 2 string 146272 146272 0.00% 148038 148038 0.00%
sqlite u32_u64_str btree_each_column 64 128 2 string 136344 136344 0.00% 138354 138354 0.00%
sqlite u32_u64_str btree_each_column 64 128 1 u64 133155 133155 0.00% 135095 135099 -0.00%

callgrind: insert bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 64 128 762831 763257 -0.06% 810097 810647 -0.07%
stdb_raw u32_u64_str btree_each_column 64 128 991397 989743 0.17% 1051091 1048797 0.22%
sqlite u32_u64_str unique_0 64 128 416908 416908 0.00% 432638 432638 0.00%
sqlite u32_u64_str btree_each_column 64 128 1023158 1023158 0.00% 1062396 1062388 0.00%

callgrind: iterate

db schema indices count total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 138453 138453 0.00% 138549 138549 0.00%
stdb_raw u32_u64_str unique_0 64 15878 15878 0.00% 15978 15978 0.00%
sqlite u32_u64_str unique_0 1024 1045786 1045786 0.00% 1049538 1049538 0.00%
sqlite u32_u64_str unique_0 64 76476 76476 0.00% 77788 77788 0.00%

callgrind: serialize_product_value

count format total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
64 json 47528 47528 0.00% 50316 50316 0.00%
64 bsatn 25509 25509 0.00% 27787 27787 0.00%
16 bsatn 8200 8200 0.00% 9628 9628 0.00%
16 json 12188 12188 0.00% 14194 14194 0.00%

callgrind: update bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 1024 17967343 17968185 -0.00% 18525177 18550015 -0.13%
stdb_raw u32_u64_str unique_0 64 128 1188745 1191002 -0.19% 1254649 1257534 -0.23%
sqlite u32_u64_str unique_0 1024 1024 1809785 1809785 0.00% 1818517 1818517 0.00%
sqlite u32_u64_str unique_0 64 128 132687 132687 0.00% 135517 135517 0.00%

@RReverser
Copy link
Member Author

benchmarks please

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 11, 2024 •
edited
Loading

Benchmarking failed. Please check the workflow run for details.

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 11, 2024 •
edited
Loading

Callgrind benchmark results

Callgrind Benchmark Report

These benchmarks were run using callgrind,
an instruction-level profiler. They allow comparisons between sqlite (sqlite), SpacetimeDB running through a module (stdb_module), and the underlying SpacetimeDB data storage engine (stdb_raw). Callgrind emulates a CPU to collect the below estimates.

Measurement changes larger than five percent are in bold.

In-memory benchmarks

callgrind: empty transaction

db total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw 6455 6455 0.00% 6555 6555 0.00%
sqlite 5609 5609 0.00% 5951 5951 0.00%

callgrind: filter

db schema indices count preload _column data_type total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str no_index 64 128 1 u64 75267 75267 0.00% 75735 75711 0.03%
stdb_raw u32_u64_str no_index 64 128 2 string 117551 117551 0.00% 118345 118321 0.02%
stdb_raw u32_u64_str btree_each_column 64 128 2 string 25071 25072 -0.00% 25629 25634 -0.02%
stdb_raw u32_u64_str btree_each_column 64 128 1 u64 24039 24039 0.00% 24467 24471 -0.02%
sqlite u32_u64_str no_index 64 128 2 string 144351 144351 0.00% 145897 145901 -0.00%
sqlite u32_u64_str no_index 64 128 1 u64 123742 123742 0.00% 125194 125194 0.00%
sqlite u32_u64_str btree_each_column 64 128 1 u64 131059 131059 0.00% 132691 132687 0.00%
sqlite u32_u64_str btree_each_column 64 128 2 string 134158 134158 0.00% 135820 135824 -0.00%

callgrind: insert bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 64 128 814751 815455 -0.09% 831991 832871 -0.11%
stdb_raw u32_u64_str btree_each_column 64 128 1037244 1040515 -0.31% 1065066 1100193 -3.19%
sqlite u32_u64_str unique_0 64 128 399360 399360 0.00% 415582 415582 0.00%
sqlite u32_u64_str btree_each_column 64 128 984611 984611 0.00% 1024095 1024099 -0.00%

callgrind: iterate

db schema indices count total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 138448 138448 0.00% 138568 138564 0.00%
stdb_raw u32_u64_str unique_0 64 15873 15873 0.00% 15985 15981 0.03%
sqlite u32_u64_str unique_0 1024 1042736 1042724 0.00% 1046126 1046110 0.00%
sqlite u32_u64_str unique_0 64 74704 74704 0.00% 75892 75892 0.00%

callgrind: serialize_product_value

count format total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
64 json 47528 47528 0.00% 50316 50316 0.00%
64 bsatn 25509 25509 0.00% 27787 27787 0.00%
16 bsatn 8200 8200 0.00% 9628 9628 0.00%
16 json 12188 12188 0.00% 14194 14194 0.00%

callgrind: update bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 1024 19472977 19061720 2.16% 20038295 19597048 2.25%
stdb_raw u32_u64_str unique_0 64 128 1250385 1251271 -0.07% 1317687 1318901 -0.09%
sqlite u32_u64_str unique_0 1024 1024 1802137 1802155 -0.00% 1811109 1811143 -0.00%
sqlite u32_u64_str unique_0 64 128 128558 128540 0.01% 131324 131306 0.01%
On-disk benchmarks

callgrind: empty transaction

db total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw 6460 6460 0.00% 6560 6564 -0.06%
sqlite 5651 5651 0.00% 6019 6019 0.00%

callgrind: filter

db schema indices count preload _column data_type total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str no_index 64 128 1 u64 75272 75272 0.00% 75704 75716 -0.02%
stdb_raw u32_u64_str no_index 64 128 2 string 117556 117548 0.01% 118226 118246 -0.02%
stdb_raw u32_u64_str btree_each_column 64 128 2 string 25076 25111 -0.14% 25606 25657 -0.20%
stdb_raw u32_u64_str btree_each_column 64 128 1 u64 24044 24044 0.00% 24460 24464 -0.02%
sqlite u32_u64_str no_index 64 128 1 u64 125663 125663 0.00% 127303 127311 -0.01%
sqlite u32_u64_str no_index 64 128 2 string 146272 146272 0.00% 148038 148038 0.00%
sqlite u32_u64_str btree_each_column 64 128 2 string 136344 136344 0.00% 138358 138354 0.00%
sqlite u32_u64_str btree_each_column 64 128 1 u64 133155 133155 0.00% 135095 135099 -0.00%

callgrind: insert bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 64 128 763494 763257 0.03% 810704 810647 0.01%
stdb_raw u32_u64_str btree_each_column 64 128 991020 989743 0.13% 1050618 1048797 0.17%
sqlite u32_u64_str unique_0 64 128 416908 416908 0.00% 432638 432638 0.00%
sqlite u32_u64_str btree_each_column 64 128 1023191 1023158 0.00% 1062421 1062388 0.00%

callgrind: iterate

db schema indices count total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 138453 138453 0.00% 138549 138549 0.00%
stdb_raw u32_u64_str unique_0 64 15878 15878 0.00% 15978 15978 0.00%
sqlite u32_u64_str unique_0 1024 1045786 1045786 0.00% 1049538 1049538 0.00%
sqlite u32_u64_str unique_0 64 76476 76476 0.00% 77788 77788 0.00%

callgrind: serialize_product_value

count format total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
64 json 47528 47528 0.00% 50316 50316 0.00%
64 bsatn 25509 25509 0.00% 27787 27787 0.00%
16 bsatn 8200 8200 0.00% 9628 9628 0.00%
16 json 12188 12188 0.00% 14194 14194 0.00%

callgrind: update bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 1024 17966413 17968185 -0.01% 18523801 18550015 -0.14%
stdb_raw u32_u64_str unique_0 64 128 1188963 1191002 -0.17% 1254883 1257534 -0.21%
sqlite u32_u64_str unique_0 1024 1024 1809785 1809785 0.00% 1818517 1818517 0.00%
sqlite u32_u64_str unique_0 64 128 132705 132687 0.01% 135551 135517 0.03%

@RReverser
Copy link
Member Author

benchmarks please

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 11, 2024 •
edited
Loading

Criterion benchmark results

Criterion benchmark report

YOU SHOULD PROBABLY IGNORE THESE RESULTS.

Criterion is a wall time based benchmarking system that is extremely noisy when run on CI. We collect these results for longitudinal analysis, but they are not reliable for comparing individual PRs.

Go look at the callgrind report instead.

empty

db on disk new latency old latency new throughput old throughput
sqlite 💿 - 410.5±1.90ns - -
sqlite 🧠 - 406.2±2.03ns - -
stdb_raw 💿 773.1±1.23ns 775.7±1.28ns - -
stdb_raw 🧠 776.7±1.54ns 773.4±0.94ns - -

insert_1

db on disk schema indices preload new latency old latency new throughput old throughput

insert_bulk

db on disk schema indices preload count new latency old latency new throughput old throughput
sqlite 💿 u32_u64_str btree_each_column 2048 256 - 592.9±1.44µs - 1686 tx/sec
sqlite 💿 u32_u64_str unique_0 2048 256 - 151.7±0.49µs - 6.4 Ktx/sec
sqlite 💿 u32_u64_u64 btree_each_column 2048 256 - 483.5±31.83µs - 2.0 Ktx/sec
sqlite 💿 u32_u64_u64 unique_0 2048 256 - 139.6±0.40µs - 7.0 Ktx/sec
sqlite 🧠 u32_u64_str btree_each_column 2048 256 - 454.6±0.87µs - 2.1 Ktx/sec
sqlite 🧠 u32_u64_str unique_0 2048 256 - 123.1±0.52µs - 7.9 Ktx/sec
sqlite 🧠 u32_u64_u64 btree_each_column 2048 256 - 369.4±0.99µs - 2.6 Ktx/sec
sqlite 🧠 u32_u64_u64 unique_0 2048 256 - 105.8±2.34µs - 9.2 Ktx/sec
stdb_raw 💿 u32_u64_str btree_each_column 2048 256 607.4±23.00µs 604.9±23.97µs 1646 tx/sec 1653 tx/sec
stdb_raw 💿 u32_u64_str unique_0 2048 256 459.6±26.29µs 487.1±14.89µs 2.1 Ktx/sec 2.0 Ktx/sec
stdb_raw 💿 u32_u64_u64 btree_each_column 2048 256 326.9±4.52µs 336.2±6.31µs 3.0 Ktx/sec 2.9 Ktx/sec
stdb_raw 💿 u32_u64_u64 unique_0 2048 256 320.6±12.91µs 353.7±8.68µs 3.0 Ktx/sec 2.8 Ktx/sec
stdb_raw 🧠 u32_u64_str btree_each_column 2048 256 303.2±0.28µs 310.0±0.46µs 3.2 Ktx/sec 3.2 Ktx/sec
stdb_raw 🧠 u32_u64_str unique_0 2048 256 190.5±0.48µs 241.0±0.19µs 5.1 Ktx/sec 4.1 Ktx/sec
stdb_raw 🧠 u32_u64_u64 btree_each_column 2048 256 235.0±0.10µs 247.7±0.09µs 4.2 Ktx/sec 3.9 Ktx/sec
stdb_raw 🧠 u32_u64_u64 unique_0 2048 256 170.0±0.47µs 220.9±0.37µs 5.7 Ktx/sec 4.4 Ktx/sec

iterate

db on disk schema indices new latency old latency new throughput old throughput
sqlite 💿 u32_u64_str unique_0 - 22.5±0.19µs - 43.4 Ktx/sec
sqlite 💿 u32_u64_u64 unique_0 - 21.4±0.10µs - 45.7 Ktx/sec
sqlite 🧠 u32_u64_str unique_0 - 19.7±0.10µs - 49.5 Ktx/sec
sqlite 🧠 u32_u64_u64 unique_0 - 18.8±0.09µs - 51.9 Ktx/sec
stdb_raw 💿 u32_u64_str unique_0 4.3±0.00µs 4.0±0.00µs 225.2 Ktx/sec 246.7 Ktx/sec
stdb_raw 💿 u32_u64_u64 unique_0 4.3±0.07µs 3.9±0.00µs 227.5 Ktx/sec 252.9 Ktx/sec
stdb_raw 🧠 u32_u64_str unique_0 4.3±0.00µs 4.0±0.01µs 224.6 Ktx/sec 247.0 Ktx/sec
stdb_raw 🧠 u32_u64_u64 unique_0 4.2±0.00µs 3.9±0.00µs 230.5 Ktx/sec 253.1 Ktx/sec

find_unique

db on disk key type preload new latency old latency new throughput old throughput

filter

db on disk key type index strategy load count new latency old latency new throughput old throughput
sqlite 💿 string index 2048 256 - 70.8±0.20µs - 13.8 Ktx/sec
sqlite 💿 u64 index 2048 256 - 67.4±0.16µs - 14.5 Ktx/sec
sqlite 🧠 string index 2048 256 - 66.7±0.10µs - 14.6 Ktx/sec
sqlite 🧠 u64 index 2048 256 - 60.9±0.07µs - 16.0 Ktx/sec
stdb_raw 💿 string index 2048 256 4.9±0.01µs 5.1±0.01µs 199.1 Ktx/sec 192.9 Ktx/sec
stdb_raw 💿 u64 index 2048 256 4.8±0.00µs 4.9±0.00µs 202.4 Ktx/sec 198.8 Ktx/sec
stdb_raw 🧠 string index 2048 256 4.9±0.01µs 5.1±0.00µs 199.2 Ktx/sec 192.9 Ktx/sec
stdb_raw 🧠 u64 index 2048 256 4.8±0.00µs 4.9±0.00µs 202.7 Ktx/sec 198.8 Ktx/sec

serialize

schema format count new latency old latency new throughput old throughput
u32_u64_str bflatn_to_bsatn_fast_path 100 3.7±0.00µs 3.6±0.00µs 26.0 Mtx/sec 26.7 Mtx/sec
u32_u64_str bflatn_to_bsatn_slow_path 100 3.5±0.01µs 2.9±0.01µs 26.9 Mtx/sec 32.4 Mtx/sec
u32_u64_str bsatn 100 17.3±0.01ns 15.3±0.01ns 5.4 Gtx/sec 6.1 Gtx/sec
u32_u64_str bsatn 100 2.1±0.01µs 2.2±0.00µs 44.5 Mtx/sec 44.2 Mtx/sec
u32_u64_str json 100 4.7±0.03µs 5.5±0.02µs 20.4 Mtx/sec 17.5 Mtx/sec
u32_u64_str json 100 8.8±0.01µs 8.7±0.04µs 10.9 Mtx/sec 10.9 Mtx/sec
u32_u64_str product_value 100 1072.5±0.48ns 1019.0±0.75ns 88.9 Mtx/sec 93.6 Mtx/sec
u32_u64_u64 bflatn_to_bsatn_fast_path 100 981.4±1.08ns 1011.4±74.52ns 97.2 Mtx/sec 94.3 Mtx/sec
u32_u64_u64 bflatn_to_bsatn_slow_path 100 2.8±0.01µs 2.4±0.01µs 34.1 Mtx/sec 39.1 Mtx/sec
u32_u64_u64 bsatn 100 15.3±0.00ns 6.8±0.01ns 6.1 Gtx/sec 13.7 Gtx/sec
u32_u64_u64 bsatn 100 1597.6±5.66ns 1535.5±2.47ns 59.7 Mtx/sec 62.1 Mtx/sec
u32_u64_u64 json 100 3.1±0.10µs 3.7±0.00µs 30.4 Mtx/sec 26.0 Mtx/sec
u32_u64_u64 json 100 5.8±0.07µs 5.9±0.35µs 16.6 Mtx/sec 16.0 Mtx/sec
u32_u64_u64 product_value 100 1074.3±1.09ns 1014.5±0.82ns 88.8 Mtx/sec 94.0 Mtx/sec
u64_u64_u32 bflatn_to_bsatn_fast_path 100 739.2±2.05ns 743.9±2.22ns 129.0 Mtx/sec 128.2 Mtx/sec
u64_u64_u32 bflatn_to_bsatn_slow_path 100 2.8±0.03µs 2.4±0.00µs 34.0 Mtx/sec 39.1 Mtx/sec
u64_u64_u32 bsatn 100 1598.3±2.72ns 1545.0±1.26ns 59.7 Mtx/sec 61.7 Mtx/sec
u64_u64_u32 bsatn 100 249.9±1.24ns 644.3±1.99ns 381.7 Mtx/sec 148.0 Mtx/sec
u64_u64_u32 json 100 3.0±0.01µs 4.0±0.01µs 31.9 Mtx/sec 23.6 Mtx/sec
u64_u64_u32 json 100 5.9±0.07µs 6.1±0.17µs 16.2 Mtx/sec 15.7 Mtx/sec
u64_u64_u32 product_value 100 1073.1±0.72ns 1014.0±0.54ns 88.9 Mtx/sec 94.1 Mtx/sec

stdb_module_large_arguments

arg size new latency old latency new throughput old throughput
64KiB - 104.9±9.63µs - -

stdb_module_print_bulk

line count new latency old latency new throughput old throughput
1 - 52.0±5.86µs - -
100 - 604.1±4.21µs - -
1000 - 3.5±0.03ms - -

remaining

name new latency old latency new throughput old throughput
special/db_game/circles/load=10 - 38.8±6.19ms - -
special/db_game/circles/load=100 - 45.3±6.01ms - -
special/db_game/csharp/circles/load=10 2.7±0.01s - - -
special/db_game/csharp/circles/load=100 2.7±0.01s - - -
special/db_game/csharp/ia_loop/load=10 351.5±0.90ms - - -
special/db_game/csharp/ia_loop/load=100 926.7±5.30ms - - -
special/db_game/ia_loop/load=500 - 147.7±1.04ms - -
special/db_game/ia_loop/load=5000 - 5.3±0.03s - -
special/db_game/rust/circles/load=10 51.4±3.24ms - - -
special/db_game/rust/circles/load=100 38.8±0.77ms - - -
special/db_game/rust/ia_loop/load=10 8.5±0.26ms - - -
special/db_game/rust/ia_loop/load=100 11.3±0.34ms - - -
special/stdb_module/csharp/large_arguments/64KiB 2.1±1.09ms - - -
special/stdb_module/csharp/print_bulk/lines=1 177.0±10.39µs - - -
special/stdb_module/csharp/print_bulk/lines=100 1922.4±180.85µs - - -
special/stdb_module/csharp/print_bulk/lines=1000 18.2±0.88ms - - -
special/stdb_module/rust/large_arguments/64KiB 95.5±9.93µs - - -
special/stdb_module/rust/print_bulk/lines=1 48.4±6.27µs - - -
special/stdb_module/rust/print_bulk/lines=100 597.4±5.10µs - - -
special/stdb_module/rust/print_bulk/lines=1000 4.6±0.94ms - - -
sqlite/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 - 54.5±0.14µs - 17.9 Ktx/sec
sqlite/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 - 47.2±0.06µs - 20.7 Ktx/sec
sqlite/🧠/update_bulk/u32_u64_str/unique_0/load=2048/count=256 - 38.4±0.12µs - 25.4 Ktx/sec
sqlite/🧠/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 - 35.6±0.20µs - 27.4 Ktx/sec
stdb_module/csharp/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 11.6±0.04ms - 86 tx/sec -
stdb_module/csharp/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 9.4±0.41ms - 106 tx/sec -
stdb_module/rust/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 1250.3±13.07µs - 799 tx/sec -
stdb_module/rust/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 967.7±16.91µs - 1033 tx/sec -
stdb_module/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 - 1256.1±8.05µs - 796 tx/sec
stdb_module/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 - 1006.3±3.13µs - 993 tx/sec
stdb_raw/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 513.9±27.06µs 648.8±16.20µs 1945 tx/sec 1541 tx/sec
stdb_raw/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 467.4±12.33µs 491.4±7.68µs 2.1 Ktx/sec 2035 tx/sec
stdb_raw/🧠/update_bulk/u32_u64_str/unique_0/load=2048/count=256 346.2±0.33µs 378.1±0.87µs 2.8 Ktx/sec 2.6 Ktx/sec
stdb_raw/🧠/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 308.3±0.26µs 342.2±0.62µs 3.2 Ktx/sec 2.9 Ktx/sec

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 11, 2024 •
edited
Loading

Callgrind benchmark results

Callgrind Benchmark Report

These benchmarks were run using callgrind,
an instruction-level profiler. They allow comparisons between sqlite (sqlite), SpacetimeDB running through a module (stdb_module), and the underlying SpacetimeDB data storage engine (stdb_raw). Callgrind emulates a CPU to collect the below estimates.

Measurement changes larger than five percent are in bold.

In-memory benchmarks

callgrind: empty transaction

db total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw 6455 6455 0.00% 6559 6555 0.06%
sqlite 5609 5609 0.00% 5951 5951 0.00%

callgrind: filter

db schema indices count preload _column data_type total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str no_index 64 128 1 u64 75267 75267 0.00% 75715 75711 0.01%
stdb_raw u32_u64_str no_index 64 128 2 string 117551 117551 0.00% 118317 118321 -0.00%
stdb_raw u32_u64_str btree_each_column 64 128 2 string 25071 25072 -0.00% 25625 25634 -0.04%
stdb_raw u32_u64_str btree_each_column 64 128 1 u64 24039 24039 0.00% 24475 24471 0.02%
sqlite u32_u64_str no_index 64 128 2 string 144351 144351 0.00% 145901 145901 0.00%
sqlite u32_u64_str no_index 64 128 1 u64 123742 123742 0.00% 125190 125194 -0.00%
sqlite u32_u64_str btree_each_column 64 128 1 u64 131059 131059 0.00% 132691 132687 0.00%
sqlite u32_u64_str btree_each_column 64 128 2 string 134158 134158 0.00% 135820 135824 -0.00%

callgrind: insert bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 64 128 814967 815455 -0.06% 832179 832871 -0.08%
stdb_raw u32_u64_str btree_each_column 64 128 1039751 1040515 -0.07% 1067569 1100193 -2.97%
sqlite u32_u64_str unique_0 64 128 399366 399360 0.00% 415588 415582 0.00%
sqlite u32_u64_str btree_each_column 64 128 984611 984611 0.00% 1024087 1024099 -0.00%

callgrind: iterate

db schema indices count total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 138448 138448 0.00% 138564 138564 0.00%
stdb_raw u32_u64_str unique_0 64 15873 15873 0.00% 15973 15981 -0.05%
sqlite u32_u64_str unique_0 1024 1042718 1042724 -0.00% 1046104 1046110 -0.00%
sqlite u32_u64_str unique_0 64 74704 74704 0.00% 75892 75892 0.00%

callgrind: serialize_product_value

count format total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
64 json 47528 47528 0.00% 50316 50316 0.00%
64 bsatn 25509 25509 0.00% 27787 27787 0.00%
16 bsatn 8200 8200 0.00% 9628 9628 0.00%
16 json 12188 12188 0.00% 14194 14194 0.00%

callgrind: update bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 1024 19058032 19061720 -0.02% 19586160 19597048 -0.06%
stdb_raw u32_u64_str unique_0 64 128 1237794 1251271 -1.08% 1305340 1318901 -1.03%
sqlite u32_u64_str unique_0 1024 1024 1802137 1802155 -0.00% 1811113 1811143 -0.00%
sqlite u32_u64_str unique_0 64 128 128540 128540 0.00% 131306 131306 0.00%
On-disk benchmarks

callgrind: empty transaction

db total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw 6460 6460 0.00% 6564 6564 0.00%
sqlite 5651 5651 0.00% 6019 6019 0.00%

callgrind: filter

db schema indices count preload _column data_type total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str no_index 64 128 1 u64 75272 75272 0.00% 75704 75716 -0.02%
stdb_raw u32_u64_str no_index 64 128 2 string 117556 117548 0.01% 118250 118246 0.00%
stdb_raw u32_u64_str btree_each_column 64 128 2 string 25076 25111 -0.14% 25602 25657 -0.21%
stdb_raw u32_u64_str btree_each_column 64 128 1 u64 24044 24044 0.00% 24460 24464 -0.02%
sqlite u32_u64_str no_index 64 128 1 u64 125663 125663 0.00% 127307 127311 -0.00%
sqlite u32_u64_str no_index 64 128 2 string 146272 146272 0.00% 148038 148038 0.00%
sqlite u32_u64_str btree_each_column 64 128 2 string 136344 136344 0.00% 138350 138354 -0.00%
sqlite u32_u64_str btree_each_column 64 128 1 u64 133155 133155 0.00% 135095 135099 -0.00%

callgrind: insert bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 64 128 763490 763257 0.03% 810820 810647 0.02%
stdb_raw u32_u64_str btree_each_column 64 128 989883 989743 0.01% 1050117 1048797 0.13%
sqlite u32_u64_str unique_0 64 128 416908 416908 0.00% 432638 432638 0.00%
sqlite u32_u64_str btree_each_column 64 128 1023164 1023158 0.00% 1062390 1062388 0.00%

callgrind: iterate

db schema indices count total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 138453 138453 0.00% 138549 138549 0.00%
stdb_raw u32_u64_str unique_0 64 15878 15878 0.00% 15978 15978 0.00%
sqlite u32_u64_str unique_0 1024 1045786 1045786 0.00% 1049538 1049538 0.00%
sqlite u32_u64_str unique_0 64 76476 76476 0.00% 77788 77788 0.00%

callgrind: serialize_product_value

count format total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
64 json 47528 47528 0.00% 50316 50316 0.00%
64 bsatn 25509 25509 0.00% 27787 27787 0.00%
16 bsatn 8200 8200 0.00% 9628 9628 0.00%
16 json 12188 12188 0.00% 14194 14194 0.00%

callgrind: update bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 1024 17965685 17968185 -0.01% 18525991 18550015 -0.13%
stdb_raw u32_u64_str unique_0 64 128 1189976 1191002 -0.09% 1256348 1257534 -0.09%
sqlite u32_u64_str unique_0 1024 1024 1809791 1809785 0.00% 1818523 1818517 0.00%
sqlite u32_u64_str unique_0 64 128 132687 132687 0.00% 135517 135517 0.00%

@RReverser RReverser closed this Dec 11, 2024
@RReverser RReverser deleted the ingvar/bench-temp-test branch December 11, 2024 17:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bfops bfops Awaiting requested review from bfops bfops will be requested when the pull request is marked ready for review bfops is a code owner

@cloutiertyler cloutiertyler Awaiting requested review from cloutiertyler cloutiertyler will be requested when the pull request is marked ready for review cloutiertyler is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@RReverser