Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: move DB Bucket schema to consumer packages #5560

Merged
merged 4 commits into from
May 30, 2023
Merged

chore: move DB Bucket schema to consumer packages #5560

merged 4 commits into from
May 30, 2023

Conversation

dapplion
Copy link
Contributor

@dapplion dapplion commented May 26, 2023 •
edited
Loading

Motivation

DB Bucket entries are all defined in the db package. Some of these entries are used in the beacon node (lodestar package), some are used in the validator (validator package).
The db package should really only be used to define the abstract db logic, and doesn't need to know which buckets exist for all downstream db consumers.

Description

Move DB Bucket schema to consumer packages

Closes #2820

@dapplion dapplion requested a review from a team as a code owner May 26, 2023 17:27
@github-actions
Copy link
Contributor

github-actions bot commented May 26, 2023 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 94a9244 Previous: 8fa4b1f Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 835.62 us/op 496.87 us/op 1.68
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 44.717 us/op 45.271 us/op 0.99
BLS verify - blst-native 1.1904 ms/op 1.2100 ms/op 0.98
BLS verifyMultipleSignatures 3 - blst-native 2.4183 ms/op 2.4895 ms/op 0.97
BLS verifyMultipleSignatures 8 - blst-native 5.1904 ms/op 5.3022 ms/op 0.98
BLS verifyMultipleSignatures 32 - blst-native 19.588 ms/op 19.255 ms/op 1.02
BLS aggregatePubkeys 32 - blst-native 25.525 us/op 25.761 us/op 0.99
BLS aggregatePubkeys 128 - blst-native 99.996 us/op 100.55 us/op 0.99
getAttestationsForBlock 63.160 ms/op 57.481 ms/op 1.10
isKnown best case - 1 super set check 258.00 ns/op 258.00 ns/op 1.00
isKnown normal case - 2 super set checks 254.00 ns/op 244.00 ns/op 1.04
isKnown worse case - 16 super set checks 256.00 ns/op 246.00 ns/op 1.04
CheckpointStateCache - add get delete 4.9280 us/op 5.0150 us/op 0.98
validate gossip signedAggregateAndProof - struct 2.7468 ms/op 2.7898 ms/op 0.98
validate gossip attestation - struct 1.3158 ms/op 1.3342 ms/op 0.99
pickEth1Vote - no votes 1.2556 ms/op 1.3232 ms/op 0.95
pickEth1Vote - max votes 9.2829 ms/op 8.4395 ms/op 1.10
pickEth1Vote - Eth1Data hashTreeRoot value x2048 8.6440 ms/op 8.6190 ms/op 1.00
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 17.205 ms/op 14.625 ms/op 1.18
pickEth1Vote - Eth1Data fastSerialize value x2048 678.85 us/op 672.46 us/op 1.01
pickEth1Vote - Eth1Data fastSerialize tree x2048 6.0746 ms/op 6.7303 ms/op 0.90
bytes32 toHexString 483.00 ns/op 456.00 ns/op 1.06
bytes32 Buffer.toString(hex) 340.00 ns/op 332.00 ns/op 1.02
bytes32 Buffer.toString(hex) from Uint8Array 541.00 ns/op 537.00 ns/op 1.01
bytes32 Buffer.toString(hex) + 0x 334.00 ns/op 351.00 ns/op 0.95
Object access 1 prop 0.16900 ns/op 0.15700 ns/op 1.08
Map access 1 prop 0.16800 ns/op 0.15500 ns/op 1.08
Object get x1000 7.2080 ns/op 6.9580 ns/op 1.04
Map get x1000 0.53300 ns/op 0.53200 ns/op 1.00
Object set x1000 50.819 ns/op 50.862 ns/op 1.00
Map set x1000 43.257 ns/op 43.491 ns/op 0.99
Return object 10000 times 0.23720 ns/op 0.23330 ns/op 1.02
Throw Error 10000 times 4.2214 us/op 4.3973 us/op 0.96
fastMsgIdFn sha256 / 200 bytes 3.4420 us/op 3.3920 us/op 1.01
fastMsgIdFn h32 xxhash / 200 bytes 280.00 ns/op 275.00 ns/op 1.02
fastMsgIdFn h64 xxhash / 200 bytes 422.00 ns/op 376.00 ns/op 1.12
fastMsgIdFn sha256 / 1000 bytes 11.780 us/op 11.148 us/op 1.06
fastMsgIdFn h32 xxhash / 1000 bytes 415.00 ns/op 400.00 ns/op 1.04
fastMsgIdFn h64 xxhash / 1000 bytes 468.00 ns/op 450.00 ns/op 1.04
fastMsgIdFn sha256 / 10000 bytes 104.81 us/op 99.885 us/op 1.05
fastMsgIdFn h32 xxhash / 10000 bytes 1.9300 us/op 1.8200 us/op 1.06
fastMsgIdFn h64 xxhash / 10000 bytes 1.3660 us/op 1.2930 us/op 1.06
enrSubnets - fastDeserialize 64 bits 1.2780 us/op 1.2560 us/op 1.02
enrSubnets - ssz BitVector 64 bits 499.00 ns/op 476.00 ns/op 1.05
enrSubnets - fastDeserialize 4 bits 176.00 ns/op 165.00 ns/op 1.07
enrSubnets - ssz BitVector 4 bits 495.00 ns/op 474.00 ns/op 1.04
prioritizePeers score -10:0 att 32-0.1 sync 2-0 103.39 us/op 103.27 us/op 1.00
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 133.61 us/op 127.37 us/op 1.05
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 165.00 us/op 159.15 us/op 1.04
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 295.67 us/op 287.54 us/op 1.03
prioritizePeers score 0:0 att 64-1 sync 4-1 352.95 us/op 345.76 us/op 1.02
array of 16000 items push then shift 1.6127 us/op 1.5552 us/op 1.04
LinkedList of 16000 items push then shift 8.8910 ns/op 8.4430 ns/op 1.05
array of 16000 items push then pop 77.965 ns/op 80.512 ns/op 0.97
LinkedList of 16000 items push then pop 8.6020 ns/op 8.1700 ns/op 1.05
array of 24000 items push then shift 2.4892 us/op 2.2626 us/op 1.10
LinkedList of 24000 items push then shift 8.8830 ns/op 8.4100 ns/op 1.06
array of 24000 items push then pop 73.984 ns/op 75.667 ns/op 0.98
LinkedList of 24000 items push then pop 8.4550 ns/op 8.1370 ns/op 1.04
intersect bitArray bitLen 8 13.346 ns/op 12.694 ns/op 1.05
intersect array and set length 8 75.418 ns/op 73.877 ns/op 1.02
intersect bitArray bitLen 128 44.182 ns/op 42.132 ns/op 1.05
intersect array and set length 128 1.0307 us/op 1.0251 us/op 1.01
Buffer.concat 32 items 2.6170 us/op 2.7750 us/op 0.94
Uint8Array.set 32 items 2.1460 us/op 2.5760 us/op 0.83
pass gossip attestations to forkchoice per slot 2.3051 ms/op 2.7388 ms/op 0.84
computeDeltas 3.0176 ms/op 2.8515 ms/op 1.06
computeProposerBoostScoreFromBalances 1.8304 ms/op 1.7247 ms/op 1.06
altair processAttestation - 250000 vs - 7PWei normalcase 2.7938 ms/op 2.0086 ms/op 1.39
altair processAttestation - 250000 vs - 7PWei worstcase 3.9579 ms/op 3.1575 ms/op 1.25
altair processAttestation - setStatus - 1/6 committees join 150.67 us/op 134.89 us/op 1.12
altair processAttestation - setStatus - 1/3 committees join 290.96 us/op 271.33 us/op 1.07
altair processAttestation - setStatus - 1/2 committees join 386.67 us/op 363.18 us/op 1.06
altair processAttestation - setStatus - 2/3 committees join 482.05 us/op 438.36 us/op 1.10
altair processAttestation - setStatus - 4/5 committees join 661.38 us/op 623.69 us/op 1.06
altair processAttestation - setStatus - 100% committees join 772.25 us/op 741.49 us/op 1.04
altair processBlock - 250000 vs - 7PWei normalcase 18.441 ms/op 18.911 ms/op 0.98
altair processBlock - 250000 vs - 7PWei normalcase hashState 27.790 ms/op 26.376 ms/op 1.05
altair processBlock - 250000 vs - 7PWei worstcase 57.686 ms/op 53.026 ms/op 1.09
altair processBlock - 250000 vs - 7PWei worstcase hashState 75.590 ms/op 67.052 ms/op 1.13
phase0 processBlock - 250000 vs - 7PWei normalcase 2.0761 ms/op 1.9368 ms/op 1.07
phase0 processBlock - 250000 vs - 7PWei worstcase 28.552 ms/op 27.696 ms/op 1.03
altair processEth1Data - 250000 vs - 7PWei normalcase 486.07 us/op 461.64 us/op 1.05
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 7.3110 us/op 6.6410 us/op 1.10
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 31.172 us/op 19.090 us/op 1.63
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 10.219 us/op 8.0240 us/op 1.27
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 9.8980 us/op 6.1730 us/op 1.60
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 112.63 us/op 74.055 us/op 1.52
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 938.70 us/op 603.56 us/op 1.56
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 909.07 us/op 891.63 us/op 1.02
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 1.3574 ms/op 876.13 us/op 1.55
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 2.3387 ms/op 2.3106 ms/op 1.01
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 2.2902 ms/op 1.4428 ms/op 1.59
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 3.9533 ms/op 3.7478 ms/op 1.05
Tree 40 250000 create 331.87 ms/op 297.79 ms/op 1.11
Tree 40 250000 get(125000) 186.37 ns/op 178.28 ns/op 1.05
Tree 40 250000 set(125000) 876.17 ns/op 922.55 ns/op 0.95
Tree 40 250000 toArray() 18.905 ms/op 16.675 ms/op 1.13
Tree 40 250000 iterate all - toArray() + loop 19.368 ms/op 16.811 ms/op 1.15
Tree 40 250000 iterate all - get(i) 67.572 ms/op 65.879 ms/op 1.03
MutableVector 250000 create 16.462 ms/op 9.6129 ms/op 1.71
MutableVector 250000 get(125000) 6.4720 ns/op 6.2910 ns/op 1.03
MutableVector 250000 set(125000) 265.62 ns/op 240.88 ns/op 1.10
MutableVector 250000 toArray() 2.7812 ms/op 2.7053 ms/op 1.03
MutableVector 250000 iterate all - toArray() + loop 2.9505 ms/op 2.7452 ms/op 1.07
MutableVector 250000 iterate all - get(i) 1.5268 ms/op 1.4801 ms/op 1.03
Array 250000 create 2.5903 ms/op 2.5456 ms/op 1.02
Array 250000 clone - spread 1.3032 ms/op 1.1507 ms/op 1.13
Array 250000 get(125000) 0.60400 ns/op 0.54200 ns/op 1.11
Array 250000 set(125000) 0.68100 ns/op 0.62400 ns/op 1.09
Array 250000 iterate all - loop 107.68 us/op 105.21 us/op 1.02
effectiveBalanceIncrements clone Uint8Array 300000 29.837 us/op 24.233 us/op 1.23
effectiveBalanceIncrements clone MutableVector 300000 424.00 ns/op 350.00 ns/op 1.21
effectiveBalanceIncrements rw all Uint8Array 300000 169.04 us/op 164.24 us/op 1.03
effectiveBalanceIncrements rw all MutableVector 300000 85.888 ms/op 77.696 ms/op 1.11
phase0 afterProcessEpoch - 250000 vs - 7PWei 116.95 ms/op 108.33 ms/op 1.08
phase0 beforeProcessEpoch - 250000 vs - 7PWei 41.540 ms/op 40.135 ms/op 1.04
altair processEpoch - mainnet_e81889 353.35 ms/op 331.92 ms/op 1.06
mainnet_e81889 - altair beforeProcessEpoch 70.513 ms/op 64.824 ms/op 1.09
mainnet_e81889 - altair processJustificationAndFinalization 18.581 us/op 18.603 us/op 1.00
mainnet_e81889 - altair processInactivityUpdates 5.9451 ms/op 5.3394 ms/op 1.11
mainnet_e81889 - altair processRewardsAndPenalties 72.636 ms/op 52.511 ms/op 1.38
mainnet_e81889 - altair processRegistryUpdates 3.5480 us/op 2.4960 us/op 1.42
mainnet_e81889 - altair processSlashings 523.00 ns/op 546.00 ns/op 0.96
mainnet_e81889 - altair processEth1DataReset 782.00 ns/op 480.00 ns/op 1.63
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.2893 ms/op 1.2226 ms/op 1.87
mainnet_e81889 - altair processSlashingsReset 7.2940 us/op 3.9860 us/op 1.83
mainnet_e81889 - altair processRandaoMixesReset 7.4850 us/op 4.7170 us/op 1.59
mainnet_e81889 - altair processHistoricalRootsUpdate 992.00 ns/op 818.00 ns/op 1.21
mainnet_e81889 - altair processParticipationFlagUpdates 3.8460 us/op 2.1070 us/op 1.83
mainnet_e81889 - altair processSyncCommitteeUpdates 849.00 ns/op 692.00 ns/op 1.23
mainnet_e81889 - altair afterProcessEpoch 114.36 ms/op 124.10 ms/op 0.92
phase0 processEpoch - mainnet_e58758 366.41 ms/op 359.22 ms/op 1.02
mainnet_e58758 - phase0 beforeProcessEpoch 175.96 ms/op 138.96 ms/op 1.27
mainnet_e58758 - phase0 processJustificationAndFinalization 25.485 us/op 17.796 us/op 1.43
mainnet_e58758 - phase0 processRewardsAndPenalties 69.194 ms/op 63.895 ms/op 1.08
mainnet_e58758 - phase0 processRegistryUpdates 12.344 us/op 8.7130 us/op 1.42
mainnet_e58758 - phase0 processSlashings 795.00 ns/op 531.00 ns/op 1.50
mainnet_e58758 - phase0 processEth1DataReset 818.00 ns/op 483.00 ns/op 1.69
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.8702 ms/op 938.41 us/op 1.99
mainnet_e58758 - phase0 processSlashingsReset 4.7920 us/op 4.3400 us/op 1.10
mainnet_e58758 - phase0 processRandaoMixesReset 7.6840 us/op 4.1800 us/op 1.84
mainnet_e58758 - phase0 processHistoricalRootsUpdate 1.0030 us/op 605.00 ns/op 1.66
mainnet_e58758 - phase0 processParticipationRecordUpdates 6.3410 us/op 3.9450 us/op 1.61
mainnet_e58758 - phase0 afterProcessEpoch 96.421 ms/op 95.843 ms/op 1.01
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.0935 ms/op 1.2349 ms/op 0.89
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.3450 ms/op 1.6506 ms/op 0.81
altair processInactivityUpdates - 250000 normalcase 28.618 ms/op 19.495 ms/op 1.47
altair processInactivityUpdates - 250000 worstcase 29.549 ms/op 26.059 ms/op 1.13
phase0 processRegistryUpdates - 250000 normalcase 10.073 us/op 6.0420 us/op 1.67
phase0 processRegistryUpdates - 250000 badcase_full_deposits 481.45 us/op 222.30 us/op 2.17
phase0 processRegistryUpdates - 250000 worstcase 0.5 136.77 ms/op 103.63 ms/op 1.32
altair processRewardsAndPenalties - 250000 normalcase 68.344 ms/op 67.549 ms/op 1.01
altair processRewardsAndPenalties - 250000 worstcase 73.330 ms/op 70.656 ms/op 1.04
phase0 getAttestationDeltas - 250000 normalcase 6.6077 ms/op 6.3696 ms/op 1.04
phase0 getAttestationDeltas - 250000 worstcase 6.4333 ms/op 6.5291 ms/op 0.99
phase0 processSlashings - 250000 worstcase 3.5354 ms/op 3.2083 ms/op 1.10
altair processSyncCommitteeUpdates - 250000 169.89 ms/op 171.97 ms/op 0.99
BeaconState.hashTreeRoot - No change 254.00 ns/op 263.00 ns/op 0.97
BeaconState.hashTreeRoot - 1 full validator 63.190 us/op 52.218 us/op 1.21
BeaconState.hashTreeRoot - 32 full validator 671.22 us/op 501.32 us/op 1.34
BeaconState.hashTreeRoot - 512 full validator 4.7744 ms/op 5.1890 ms/op 0.92
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 63.677 us/op 61.918 us/op 1.03
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 845.35 us/op 899.60 us/op 0.94
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 12.481 ms/op 11.236 ms/op 1.11
BeaconState.hashTreeRoot - 1 balances 50.388 us/op 49.318 us/op 1.02
BeaconState.hashTreeRoot - 32 balances 548.70 us/op 451.00 us/op 1.22
BeaconState.hashTreeRoot - 512 balances 5.3621 ms/op 4.0005 ms/op 1.34
BeaconState.hashTreeRoot - 250000 balances 78.142 ms/op 75.999 ms/op 1.03
aggregationBits - 2048 els - zipIndexesInBitList 15.456 us/op 16.208 us/op 0.95
regular array get 100000 times 32.550 us/op 32.878 us/op 0.99
wrappedArray get 100000 times 32.560 us/op 32.686 us/op 1.00
arrayWithProxy get 100000 times 15.878 ms/op 15.521 ms/op 1.02
ssz.Root.equals 536.00 ns/op 675.00 ns/op 0.79
byteArrayEquals 531.00 ns/op 622.00 ns/op 0.85
shuffle list - 16384 els 6.6379 ms/op 6.7372 ms/op 0.99
shuffle list - 250000 els 97.731 ms/op 98.946 ms/op 0.99
processSlot - 1 slots 8.9730 us/op 8.4110 us/op 1.07
processSlot - 32 slots 1.3132 ms/op 1.3078 ms/op 1.00
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 41.745 ms/op 37.263 ms/op 1.12
getCommitteeAssignments - req 1 vs - 250000 vc 2.9049 ms/op 2.8954 ms/op 1.00
getCommitteeAssignments - req 100 vs - 250000 vc 4.1187 ms/op 4.1052 ms/op 1.00
getCommitteeAssignments - req 1000 vs - 250000 vc 4.4373 ms/op 4.3661 ms/op 1.02
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.5300 ns/op 4.5000 ns/op 1.01
state getBlockRootAtSlot - 250000 vs - 7PWei 849.60 ns/op 968.48 ns/op 0.88
computeProposers - vc 250000 10.418 ms/op 10.504 ms/op 0.99
computeEpochShuffling - vc 250000 101.65 ms/op 100.04 ms/op 1.02
getNextSyncCommittee - vc 250000 176.36 ms/op 169.10 ms/op 1.04
computeSigningRoot for AttestationData 14.250 us/op 13.451 us/op 1.06
hash AttestationData serialized data then Buffer.toString(base64) 2.4136 us/op 2.4152 us/op 1.00
toHexString serialized data 1.0748 us/op 1.0430 us/op 1.03
Buffer.toString(base64) 317.31 ns/op 319.96 ns/op 0.99

by benchmarkbot/action

@dapplion dapplion changed the title Move DB Bucket schema to consumer packages chore: move DB Bucket schema to consumer packages May 29, 2023
@wemeetagain wemeetagain merged commit 78bb300 into unstable May 30, 2023
@wemeetagain wemeetagain deleted the dapplion/bucket-def branch May 30, 2023 14:38
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.9.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Move DB Bucket definitions to consumer packages
2 participants
@dapplion @wemeetagain