perf: Add badger multi scan support #85

AndrewSisley · 2021-12-09T16:07:42Z

Closes #60

Adds support for iterating through multiple badger prefixes and defines the interfaces for use by other datastores.

Tested the new shim by flipping the tests back to the Map datastore - found two production bugs in doing so (fixed with commits Move discard to after error check and Fix if-else).

Suggest not merging until bench marked - is a decent sized change, and it also disables badger's PrefetchValues - the performance impact of which I do not know.

Wait until benchmarks are in
Suggest merging after the key refactoring, as this is the smaller/more contained PR and it should be safer/easier that way around

jsimnz

LGTM.

I had some questions/issues a few days ago when I first reviewed, but either I missed some files/changes or I was just tired as It all make sense now!

Will need to rebase and resolve any (potential) linter issues

AndrewSisley · 2021-12-23T15:36:24Z

LGTM.

I had some questions/issues a few days ago when I first reviewed, but either I missed some files/changes or I was just tired as It all make sense now!

Will need to rebase and resolve any (potential) linter issues

Nice cheers John :)

jsimnz · 2022-01-23T18:54:33Z

Just to confirm, this is blocked by benchmark stuff correct @AndrewSisley ?

AndrewSisley · 2022-01-23T19:50:48Z

Just to confirm, this is blocked by benchmark stuff correct @AndrewSisley ?

Correct - it changes some badgerDb params which might impact performance

codecov · 2022-01-26T18:05:36Z

Codecov Report

Merging #85 (b93bc96) into develop (846122b) will decrease coverage by 0.34%.
The diff coverage is 54.91%.

@@             Coverage Diff             @@
##           develop      #85      +/-   ##
===========================================
- Coverage    59.06%   58.71%   -0.35%     
===========================================
  Files           87       89       +2     
  Lines         8344     8531     +187     
===========================================
+ Hits          4928     5009      +81     
- Misses        2890     2980      +90     
- Partials       526      542      +16

Impacted Files	Coverage Δ
db/collection_update.go	`40.22% <28.57%> (-0.14%)`	⬇️
db/collection_get.go	`53.33% <33.33%> (-6.13%)`	⬇️
datastores/badger/v3/iterator.go	`51.79% <51.79%> (ø)`
datastores/badger/v3/datastore.go	`32.84% <57.14%> (-0.71%)`	⬇️
db/txn.go	`56.17% <57.14%> (-8.34%)`	⬇️
datastores/iterable/iterable_transaction_shim.go	`67.85% <67.85%> (ø)`
db/fetcher/fetcher.go	`63.43% <75.00%> (-1.88%)`	⬇️

AndrewSisley · 2022-02-01T21:58:47Z

Locally run benchmarks for dev branch:

goos: linux
goarch: amd64
pkg: github.com/sourcenetwork/defradb/bench/collection
cpu: Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz
Benchmark_Collection_UserSimple_CreateMany_Sync_0_10-8             9     239646252 ns/op
Benchmark_Collection_UserSimple_CreateMany_Sync_0_100-8            1    2713297484 ns/op
Benchmark_Collection_UserSimple_Create_Sync_0_1-8                 66      20276092 ns/op
Benchmark_Collection_UserSimple_Create_Sync_0_10-8                10     225022955 ns/op
Benchmark_Collection_UserSimple_Create_Sync_0_100-8                1    2217967837 ns/op
Benchmark_Collection_UserSimple_Create_Sync_0_1000-8               1    29229935496 ns/op
Benchmark_Collection_UserSimple_Create_Async_0_1-8                66      20609855 ns/op
Benchmark_Collection_UserSimple_Create_Async_0_100-8               2    1326814788 ns/op
Benchmark_Collection_UserSimple_Create_Async_0_1000-8              1    16803631883 ns/op

...

Benchmark_Planner_UserSimple_ParseQuery-8           2466        409320 ns/op
Benchmark_Planner_UserSimple_MakePlan-8             5420        192085 ns/op
PASS
ok      github.com/sourcenetwork/defradb/bench/query/planner    3.343s
goos: linux
goarch: amd64
pkg: github.com/sourcenetwork/defradb/bench/query/simple
cpu: Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz
Benchmark_Query_UserSimple_Query_Sync_1-8                                555       2071934 ns/op
Benchmark_Query_UserSimple_Query_Sync_10-8                               121       9383499 ns/op
Benchmark_Query_UserSimple_Query_Sync_100-8                               15      72842972 ns/op
Benchmark_Query_UserSimple_Query_Sync_1000-8                               2     871541347 ns/op
Benchmark_Query_UserSimple_Query_WithFilter_Sync_1-8                     339       3006475 ns/op
Benchmark_Query_UserSimple_Query_WithFilter_Sync_10-8                     87      13092536 ns/op
Benchmark_Query_UserSimple_Query_WithFilter_Sync_100-8                    13      88203815 ns/op
Benchmark_Query_UserSimple_Query_WithFilter_Sync_1000-8                    2     917211043 ns/op
Benchmark_Query_UserSimple_Query_WithLimitOffset_Sync_1-8                337       3003535 ns/op
Benchmark_Query_UserSimple_Query_WithLimitOffset_Sync_10-8                85      13379456 ns/op
Benchmark_Query_UserSimple_Query_WithLimitOffset_Sync_100-8               50      21518782 ns/op
Benchmark_Query_UserSimple_Query_WithLimitOffset_Sync_1000-8              43      26146472 ns/op
Benchmark_Query_UserSimple_Query_WithMultiLookup_Sync_10-8                14      81920500 ns/op
Benchmark_Query_UserSimple_Query_WithMultiLookup_Sync_100-8               12      90098140 ns/op
Benchmark_Query_UserSimple_Query_WithMultiLookup_Sync_1000-8              10     107609424 ns/op
Benchmark_Query_UserSimple_Query_WithSingleLookup_Sync_1-8               330       3104380 ns/op
Benchmark_Query_UserSimple_Query_WithSingleLookup_Sync_10-8              135       9382646 ns/op
Benchmark_Query_UserSimple_Query_WithSingleLookup_Sync_100-8             103       9777702 ns/op
Benchmark_Query_UserSimple_Query_WithSingleLookup_Sync_1000-8             86      12409382 ns/op
Benchmark_Query_UserSimple_Query_WithSort_Sync_1-8                       315       3229584 ns/op
Benchmark_Query_UserSimple_Query_WithSort_Sync_10-8                       81      14130994 ns/op
Benchmark_Query_UserSimple_Query_WithSort_Sync_100-8                      12      97549490 ns/op
Benchmark_Query_UserSimple_Query_WithSort_Sync_1000-8                      1    1096847379 ns/op
PASS
ok      github.com/sourcenetwork/defradb/bench/query/simple 443.801s
goos: linux
goarch: amd64
pkg: github.com/sourcenetwork/defradb/bench/storage
cpu: Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:0064-8                94017         12190 ns/op
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:0128-8                96955         12718 ns/op
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:0256-8                90009         13115 ns/op
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:0512-8                85572         13771 ns/op
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:1024-8                72784         16071 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:0064-8                8890        121504 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:0128-8                8868        125167 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:0256-8                7738        131765 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:0512-8                7972        134999 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:1024-8                6601        163579 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:0064-8                961       1164513 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:0128-8                921       1232905 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:0256-8                915       1280763 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:0512-8                867       1343811 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:1024-8                744       1593338 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:0064-8              80648         13615 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:0128-8              82878         14526 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:0256-8              79682         15135 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:0512-8              74599         16309 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:1024-8              65233         18405 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:0064-8              7707        140614 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:0128-8              7513        146161 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:0256-8              7485        152235 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:0512-8              6416        156983 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:1024-8              5950        179254 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:0064-8              831       1446060 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:0128-8              843       1495171 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:0256-8              754       1472171 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:0512-8              741       1622043 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:1024-8              663       1856769 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:0064-8            6513        177718 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:0128-8            6414        179410 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:0256-8            6159        188714 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:0512-8            5709        202788 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:1024-8            4972        233545 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:0064-8           2718        444720 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:0128-8           2509        476643 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:0256-8           2188        545880 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:0512-8           1800        665289 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:1024-8           1292        935881 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:0064-8           411       2862848 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:0128-8           374       3200378 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:0256-8           310       3728676 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:0512-8           235       4946712 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:1024-8           142       7548855 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:0064-8          6398        177205 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:0128-8          6342        181607 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:0256-8          6138        187895 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:0512-8          5652        203607 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:1024-8          4902        236237 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:0064-8         2697        449182 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:0128-8         2496        508059 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:0256-8         2200        544070 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:0512-8         1797        681540 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:1024-8         1276        971957 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:0064-8                 380       3206424 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:0128-8                 356       3692331 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:0256-8                 312       3725360 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:0512-8                 229       4922088 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:1024-8                 150       7758769 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:0064-8                        9090        125182 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:0128-8                        8908        128488 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:0256-8                        8446        136146 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:0512-8                        7729        150792 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:1024-8                        6522        177592 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:0064-8                        951       1250454 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:0128-8                        936       1288475 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:0256-8                        882       1374813 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:0512-8                        802       1502091 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:1024-8                        680       1765798 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:0064-8                        93      12463592 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:0128-8                        91      12953270 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:0256-8                        86      13495263 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:0512-8                        78      14958467 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:1024-8                        66      17697077 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:0064-8                      9069        124922 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:0128-8                      8817        128710 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:0256-8                      8448        134179 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:0512-8                      7563        151317 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:1024-8                      6318        178023 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:0064-8                      962       1256876 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:0128-8                      931       1288262 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:0256-8                      884       1363950 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:0512-8                      795       1496059 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:1024-8                      674       1775000 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:0064-8                      92      12548428 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:0128-8                      90      12941666 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:0256-8                      85      13646710 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:0512-8                      78      14931087 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:1024-8                      64      17853992 ns/op

AndrewSisley · 2022-02-01T21:59:33Z

Locally run benchmarks for this branch:

goos: linux
goarch: amd64
pkg: github.com/sourcenetwork/defradb/bench/collection
cpu: Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz
Benchmark_Collection_UserSimple_CreateMany_Sync_0_10-8             9     243187180 ns/op
Benchmark_Collection_UserSimple_CreateMany_Sync_0_100-8            1    2745112830 ns/op
Benchmark_Collection_UserSimple_Create_Sync_0_1-8                 62      20346360 ns/op
Benchmark_Collection_UserSimple_Create_Sync_0_10-8                10     226589456 ns/op
Benchmark_Collection_UserSimple_Create_Sync_0_100-8                1    2250266856 ns/op
Benchmark_Collection_UserSimple_Create_Sync_0_1000-8               1    29799735746 ns/op
Benchmark_Collection_UserSimple_Create_Async_0_1-8                60      19167256 ns/op
Benchmark_Collection_UserSimple_Create_Async_0_100-8               2    1416501997 ns/op
Benchmark_Collection_UserSimple_Create_Async_0_1000-8              1    17324384707 ns/op

...

Benchmark_Planner_UserSimple_ParseQuery-8           2544        412880 ns/op
Benchmark_Planner_UserSimple_MakePlan-8             5996        207696 ns/op
PASS
ok      github.com/sourcenetwork/defradb/bench/query/planner    3.582s
goos: linux
goarch: amd64
pkg: github.com/sourcenetwork/defradb/bench/query/simple
cpu: Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz
Benchmark_Query_UserSimple_Query_Sync_1-8                                894       1270251 ns/op
Benchmark_Query_UserSimple_Query_Sync_10-8                               373       2699876 ns/op
Benchmark_Query_UserSimple_Query_Sync_100-8                               50      22466270 ns/op
Benchmark_Query_UserSimple_Query_Sync_1000-8                               3     403098025 ns/op
Benchmark_Query_UserSimple_Query_WithFilter_Sync_1-8                     585       2027252 ns/op
Benchmark_Query_UserSimple_Query_WithFilter_Sync_10-8                    211       5021560 ns/op
Benchmark_Query_UserSimple_Query_WithFilter_Sync_100-8                    33      38604404 ns/op
Benchmark_Query_UserSimple_Query_WithFilter_Sync_1000-8                    3     477738574 ns/op
Benchmark_Query_UserSimple_Query_WithLimitOffset_Sync_1-8                565       2133137 ns/op
Benchmark_Query_UserSimple_Query_WithLimitOffset_Sync_10-8               195       5496067 ns/op
Benchmark_Query_UserSimple_Query_WithLimitOffset_Sync_100-8              160       9156632 ns/op
Benchmark_Query_UserSimple_Query_WithLimitOffset_Sync_1000-8              92      11406405 ns/op
Benchmark_Query_UserSimple_Query_WithMultiLookup_Sync_10-8                86      13351185 ns/op
Benchmark_Query_UserSimple_Query_WithMultiLookup_Sync_100-8               86      13503928 ns/op
Benchmark_Query_UserSimple_Query_WithMultiLookup_Sync_1000-8              64      19924133 ns/op
Benchmark_Query_UserSimple_Query_WithSingleLookup_Sync_1-8               561       2107686 ns/op
Benchmark_Query_UserSimple_Query_WithSingleLookup_Sync_10-8              548       2153916 ns/op
Benchmark_Query_UserSimple_Query_WithSingleLookup_Sync_100-8             528       2267160 ns/op
Benchmark_Query_UserSimple_Query_WithSingleLookup_Sync_1000-8            344       2974032 ns/op
Benchmark_Query_UserSimple_Query_WithSort_Sync_1-8                       537       2214274 ns/op
Benchmark_Query_UserSimple_Query_WithSort_Sync_10-8                      189       5779045 ns/op
Benchmark_Query_UserSimple_Query_WithSort_Sync_100-8                      28      44124662 ns/op
Benchmark_Query_UserSimple_Query_WithSort_Sync_1000-8                      3     464295772 ns/op
PASS
ok      github.com/sourcenetwork/defradb/bench/query/simple 560.847s
goos: linux
goarch: amd64
pkg: github.com/sourcenetwork/defradb/bench/storage
cpu: Intel(R) Core(TM) i5-8300H CPU @ 2.30GHz
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:0064-8                88603         12552 ns/op
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:0128-8                94545         12933 ns/op
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:0256-8                90648         13060 ns/op
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:0512-8                85755         13612 ns/op
Benchmark_Storage_Simple_Read_Sync_1_1/ValueSize:1024-8                67999         16305 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:0064-8                8994        121910 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:0128-8                9025        123073 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:0256-8                8832        136605 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:0512-8                7654        152236 ns/op
Benchmark_Storage_Simple_Read_Sync_1_10/ValueSize:1024-8                6014        188976 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:0064-8                916       1326292 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:0128-8                928       1303539 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:0256-8                908       1308950 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:0512-8                807       1464784 ns/op
Benchmark_Storage_Simple_Read_Sync_1_100/ValueSize:1024-8                660       1573795 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:0064-8              82942         14316 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:0128-8              84385         15118 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:0256-8              76015         14880 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:0512-8              63942         16495 ns/op
Benchmark_Storage_Simple_Read_Sync_100_1/ValueSize:1024-8              61354         19670 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:0064-8              8212        141801 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:0128-8              7772        141227 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:0256-8              7471        151593 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:0512-8              6682        158443 ns/op
Benchmark_Storage_Simple_Read_Sync_100_10/ValueSize:1024-8              6034        185481 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:0064-8              889       1387460 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:0128-8              829       1498827 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:0256-8              771       1523385 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:0512-8              739       1591666 ns/op
Benchmark_Storage_Simple_Read_Sync_100_100/ValueSize:1024-8              645       1821382 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:0064-8            6523        178796 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:0128-8            6397        182121 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:0256-8            6194        188739 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:0512-8            5727        204451 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_1/ValueSize:1024-8            4935        236104 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:0064-8           2698        444945 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:0128-8           2504        478294 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:0256-8           2166        543099 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:0512-8           1808        663470 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_10/ValueSize:1024-8           1305        959012 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:0064-8           415       2969197 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:0128-8           385       3832823 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:0256-8           308       5076291 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:0512-8           234       5252384 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_0_100/ValueSize:1024-8           154       7935671 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:0064-8          6444        178154 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:0128-8          6412        178915 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:0256-8          6150        187737 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:0512-8          5683        203650 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_1/ValueSize:1024-8          4875        234201 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:0064-8         2682        446590 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:0128-8         2515        477534 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:0256-8         2233        545660 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:0512-8         1807        665783 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_10/ValueSize:1024-8         1252       1140755 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:0064-8                 414       2941438 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:0128-8                 289       3491351 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:0256-8                 302       3690479 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:0512-8                 231       5119448 ns/op
Benchmark_Storage_Simple_WriteMany_Sync_100_100/ValueSize:1024-8                 150       7476156 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:0064-8                        9192        124358 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:0128-8                        8840        128151 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:0256-8                        8532        135479 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:0512-8                        7621        155178 ns/op
Benchmark_Storage_Simple_Write_Sync_0_1/ValueSize:1024-8                        6462        189510 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:0064-8                        872       1296895 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:0128-8                        930       1280023 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:0256-8                        885       1363668 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:0512-8                        804       1491563 ns/op
Benchmark_Storage_Simple_Write_Sync_0_10/ValueSize:1024-8                        680       1771564 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:0064-8                        92      13618863 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:0128-8                        88      13306622 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:0256-8                        76      13597377 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:0512-8                        78      14984962 ns/op
Benchmark_Storage_Simple_Write_Sync_0_100/ValueSize:1024-8                        66      17747707 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:0064-8                      9098        124415 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:0128-8                      8817        128719 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:0256-8                      8449        137050 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:0512-8                      7533        150149 ns/op
Benchmark_Storage_Simple_Write_Sync_100_1/ValueSize:1024-8                      6367        177450 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:0064-8                      960       1246423 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:0128-8                      927       1285406 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:0256-8                      886       1349290 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:0512-8                      798       1496086 ns/op
Benchmark_Storage_Simple_Write_Sync_100_10/ValueSize:1024-8                      668       1771530 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:0064-8                      93      12432176 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:0128-8                      90      13064219 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:0256-8                      82      13597134 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:0512-8                      79      14926321 ns/op
Benchmark_Storage_Simple_Write_Sync_100_100/ValueSize:1024-8                      64      17810755 ns/op

jsimnz

Looks good, minor issue with the iterable naming / package organization (details in the comments).

I think the mutex is overkill to protect against the race condition, but we can track that in a diff issue.

datastores/badger/v3/iterator.go

client/core.go

datastores/badger/v3/iterator.go

jsimnz

LGTM. Awesome stuff, and the potential speedup is really encouraging!

len=1 handled in MergeAscending, no need for the extra code-branch

Lines are unchanged, not sure why flagged now

* Close query when updating * Close fetcher on collection get * Simplify spans logic len=1 handled in MergeAscending, no need for the extra code-branch * Add proper multi-scan support for badger transactions * Fix linter errors Lines are unchanged, not sure why flagged now

AndrewSisley added bug Something isn't working area/datastore Related to the datastore / storage engine system perf Performance issue or suggestion labels Dec 9, 2021

AndrewSisley requested review from jsimnz and shahzadlone December 9, 2021 16:07

AndrewSisley self-assigned this Dec 9, 2021

AndrewSisley linked an issue Dec 9, 2021 that may be closed by this pull request

Support multi spans and on Document Fetcher. #60

Closed

jsimnz requested changes Dec 22, 2021

View reviewed changes

jsimnz added the blocked label Jan 24, 2022

AndrewSisley force-pushed the sisley/perf/I60-badger-multi-scan branch 3 times, most recently from 74f67a3 to 2038ce4 Compare January 26, 2022 18:03

AndrewSisley force-pushed the sisley/perf/I60-badger-multi-scan branch 2 times, most recently from 4979346 to 3fbe92c Compare February 1, 2022 21:54

AndrewSisley requested a review from jsimnz February 1, 2022 22:21

jsimnz requested changes Feb 2, 2022

View reviewed changes

datastores/badger/v3/iterator.go Outdated Show resolved Hide resolved

client/core.go Outdated Show resolved Hide resolved

datastores/badger/v3/iterator.go Show resolved Hide resolved

AndrewSisley force-pushed the sisley/perf/I60-badger-multi-scan branch 3 times, most recently from 54269ea to a79d683 Compare February 2, 2022 04:06

AndrewSisley requested a review from jsimnz February 2, 2022 04:07

jsimnz approved these changes Feb 2, 2022

View reviewed changes

AndrewSisley added 3 commits February 2, 2022 00:26

Close query when updating

a4dcdeb

Close fetcher on collection get

051708d

Simplify spans logic

f09217d

len=1 handled in MergeAscending, no need for the extra code-branch

AndrewSisley added 2 commits February 2, 2022 00:28

Add proper multi-scan support for badger transactions

2c3d335

Fix linter errors

b93bc96

Lines are unchanged, not sure why flagged now

AndrewSisley force-pushed the sisley/perf/I60-badger-multi-scan branch from a79d683 to b93bc96 Compare February 2, 2022 05:28

AndrewSisley merged commit 4e88e92 into develop Feb 2, 2022

AndrewSisley deleted the sisley/perf/I60-badger-multi-scan branch February 2, 2022 05:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: Add badger multi scan support #85

perf: Add badger multi scan support #85

AndrewSisley commented Dec 9, 2021 •

edited

Loading

jsimnz left a comment

AndrewSisley commented Dec 23, 2021

jsimnz commented Jan 23, 2022

AndrewSisley commented Jan 23, 2022

codecov bot commented Jan 26, 2022 •

edited

Loading

AndrewSisley commented Feb 1, 2022

AndrewSisley commented Feb 1, 2022

jsimnz left a comment

jsimnz left a comment

perf: Add badger multi scan support #85

perf: Add badger multi scan support #85

Conversation

AndrewSisley commented Dec 9, 2021 • edited Loading

jsimnz left a comment

Choose a reason for hiding this comment

AndrewSisley commented Dec 23, 2021

jsimnz commented Jan 23, 2022

AndrewSisley commented Jan 23, 2022

codecov bot commented Jan 26, 2022 • edited Loading

Codecov Report

AndrewSisley commented Feb 1, 2022

AndrewSisley commented Feb 1, 2022

jsimnz left a comment

Choose a reason for hiding this comment

jsimnz left a comment

Choose a reason for hiding this comment

AndrewSisley commented Dec 9, 2021 •

edited

Loading

codecov bot commented Jan 26, 2022 •

edited

Loading