Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(slashing-protection): filter by db bucket when loading pubkeys #5437

Merged
merged 2 commits into from
May 4, 2023
Merged

fix(slashing-protection): filter by db bucket when loading pubkeys #5437

merged 2 commits into from
May 4, 2023

Conversation

nflaig
Copy link
Member

@nflaig nflaig commented Apr 29, 2023

Motivation

Closes #4261

Description

Applies min-max key filter to db query to only get pubkeys from bucket used by repository. The OOM mentioned in #4261 was caused by a combination of #5356 (causing the db state to be quite huge) and the db query not applying any filtering.

Note: There is currently no limit on how many attestation are stored in the db. In some extrem cases could still run into OOM issues. This is likely only relevant for big operators that run many validators but pruning strategy for attestation would be something useful to implement in the future.

lodestar/packages/validator/src/slashingProtection/attestation/index.ts

Line 46 in f0dec8c

// TODO: Implement safe clean-up of stored attestations

Other solution that can be implemented is to provide the option to just export the slashing protection for a subset of validators

lodestar/packages/cli/src/cmds/validator/slashingProtection/export.ts

Line 49 in f0dec8c

// TODO: Allow format version and pubkeys to be customized with CLI args

Depends on #5436 to be merged first to avoid conflicts.

@nflaig nflaig requested a review from a team as a code owner April 29, 2023 13:47
@nflaig nflaig force-pushed the nflaig/vc-db-metrics-bucket-labels branch from 15351bf to c38e951 Compare April 29, 2023 14:33
@nflaig nflaig force-pushed the nflaig/apply-min-max-key-filtering branch from d223c7b to 0602ee5 Compare April 29, 2023 14:38
Base automatically changed from nflaig/vc-db-metrics-bucket-labels to unstable April 29, 2023 16:51
@nflaig nflaig force-pushed the nflaig/apply-min-max-key-filtering branch from 0602ee5 to f1aaade Compare April 29, 2023 16:54
@github-actions
Copy link
Contributor

github-actions bot commented Apr 29, 2023 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 2d4bd20 Previous: 2fbf178 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 860.84 us/op 505.54 us/op 1.70
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 46.285 us/op 46.593 us/op 0.99
BLS verify - blst-native 1.2092 ms/op 1.2273 ms/op 0.99
BLS verifyMultipleSignatures 3 - blst-native 2.4594 ms/op 2.4959 ms/op 0.99
BLS verifyMultipleSignatures 8 - blst-native 5.3259 ms/op 5.3762 ms/op 0.99
BLS verifyMultipleSignatures 32 - blst-native 19.037 ms/op 19.383 ms/op 0.98
BLS aggregatePubkeys 32 - blst-native 25.610 us/op 25.814 us/op 0.99
BLS aggregatePubkeys 128 - blst-native 99.787 us/op 99.982 us/op 1.00
getAttestationsForBlock 53.161 ms/op 53.235 ms/op 1.00
isKnown best case - 1 super set check 247.00 ns/op 251.00 ns/op 0.98
isKnown normal case - 2 super set checks 244.00 ns/op 243.00 ns/op 1.00
isKnown worse case - 16 super set checks 245.00 ns/op 243.00 ns/op 1.01
CheckpointStateCache - add get delete 4.5840 us/op 4.9180 us/op 0.93
validate gossip signedAggregateAndProof - struct 2.7519 ms/op 2.7750 ms/op 0.99
validate gossip attestation - struct 1.3071 ms/op 1.3314 ms/op 0.98
pickEth1Vote - no votes 1.2292 ms/op 1.2078 ms/op 1.02
pickEth1Vote - max votes 11.030 ms/op 10.758 ms/op 1.03
pickEth1Vote - Eth1Data hashTreeRoot value x2048 8.9265 ms/op 8.8227 ms/op 1.01
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 14.029 ms/op 15.265 ms/op 0.92
pickEth1Vote - Eth1Data fastSerialize value x2048 607.57 us/op 673.27 us/op 0.90
pickEth1Vote - Eth1Data fastSerialize tree x2048 7.3507 ms/op 7.9822 ms/op 0.92
bytes32 toHexString 464.00 ns/op 489.00 ns/op 0.95
bytes32 Buffer.toString(hex) 326.00 ns/op 355.00 ns/op 0.92
bytes32 Buffer.toString(hex) from Uint8Array 539.00 ns/op 556.00 ns/op 0.97
bytes32 Buffer.toString(hex) + 0x 329.00 ns/op 359.00 ns/op 0.92
Object access 1 prop 0.15300 ns/op 0.15900 ns/op 0.96
Map access 1 prop 0.15600 ns/op 0.16900 ns/op 0.92
Object get x1000 6.9010 ns/op 6.5060 ns/op 1.06
Map get x1000 0.50500 ns/op 0.62300 ns/op 0.81
Object set x1000 48.710 ns/op 54.171 ns/op 0.90
Map set x1000 40.875 ns/op 44.249 ns/op 0.92
Return object 10000 times 0.22620 ns/op 0.23720 ns/op 0.95
Throw Error 10000 times 4.1297 us/op 4.1253 us/op 1.00
fastMsgIdFn sha256 / 200 bytes 3.3180 us/op 3.3610 us/op 0.99
fastMsgIdFn h32 xxhash / 200 bytes 264.00 ns/op 277.00 ns/op 0.95
fastMsgIdFn h64 xxhash / 200 bytes 367.00 ns/op 396.00 ns/op 0.93
fastMsgIdFn sha256 / 1000 bytes 11.244 us/op 11.548 us/op 0.97
fastMsgIdFn h32 xxhash / 1000 bytes 402.00 ns/op 403.00 ns/op 1.00
fastMsgIdFn h64 xxhash / 1000 bytes 450.00 ns/op 480.00 ns/op 0.94
fastMsgIdFn sha256 / 10000 bytes 101.00 us/op 101.90 us/op 0.99
fastMsgIdFn h32 xxhash / 10000 bytes 1.8020 us/op 1.9610 us/op 0.92
fastMsgIdFn h64 xxhash / 10000 bytes 1.2730 us/op 1.4230 us/op 0.89
enrSubnets - fastDeserialize 64 bits 1.2560 us/op 1.3060 us/op 0.96
enrSubnets - ssz BitVector 64 bits 473.00 ns/op 479.00 ns/op 0.99
enrSubnets - fastDeserialize 4 bits 166.00 ns/op 170.00 ns/op 0.98
enrSubnets - ssz BitVector 4 bits 468.00 ns/op 497.00 ns/op 0.94
prioritizePeers score -10:0 att 32-0.1 sync 2-0 101.10 us/op 106.13 us/op 0.95
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 125.23 us/op 142.60 us/op 0.88
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 161.08 us/op 168.70 us/op 0.95
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 288.48 us/op 303.86 us/op 0.95
prioritizePeers score 0:0 att 64-1 sync 4-1 349.59 us/op 374.06 us/op 0.93
array of 16000 items push then shift 1.5600 us/op 1.5866 us/op 0.98
LinkedList of 16000 items push then shift 8.4900 ns/op 8.8680 ns/op 0.96
array of 16000 items push then pop 76.889 ns/op 86.268 ns/op 0.89
LinkedList of 16000 items push then pop 8.0400 ns/op 8.3590 ns/op 0.96
array of 24000 items push then shift 2.2324 us/op 2.3805 us/op 0.94
LinkedList of 24000 items push then shift 8.3060 ns/op 8.8170 ns/op 0.94
array of 24000 items push then pop 75.134 ns/op 74.273 ns/op 1.01
LinkedList of 24000 items push then pop 7.9630 ns/op 8.3900 ns/op 0.95
intersect bitArray bitLen 8 12.569 ns/op 13.187 ns/op 0.95
intersect array and set length 8 73.366 ns/op 74.780 ns/op 0.98
intersect bitArray bitLen 128 41.589 ns/op 43.549 ns/op 0.95
intersect array and set length 128 993.56 ns/op 998.48 ns/op 1.00
Buffer.concat 32 items 2.7500 us/op 2.4840 us/op 1.11
Uint8Array.set 32 items 2.7650 us/op 2.0990 us/op 1.32
pass gossip attestations to forkchoice per slot 2.6650 ms/op 2.7416 ms/op 0.97
computeDeltas 2.8300 ms/op 3.3479 ms/op 0.85
computeProposerBoostScoreFromBalances 1.7855 ms/op 1.8413 ms/op 0.97
altair processAttestation - 250000 vs - 7PWei normalcase 2.1171 ms/op 2.1482 ms/op 0.99
altair processAttestation - 250000 vs - 7PWei worstcase 3.2518 ms/op 3.3068 ms/op 0.98
altair processAttestation - setStatus - 1/6 committees join 136.12 us/op 139.47 us/op 0.98
altair processAttestation - setStatus - 1/3 committees join 269.44 us/op 270.49 us/op 1.00
altair processAttestation - setStatus - 1/2 committees join 359.98 us/op 364.75 us/op 0.99
altair processAttestation - setStatus - 2/3 committees join 455.20 us/op 462.56 us/op 0.98
altair processAttestation - setStatus - 4/5 committees join 638.89 us/op 649.19 us/op 0.98
altair processAttestation - setStatus - 100% committees join 745.22 us/op 729.59 us/op 1.02
altair processBlock - 250000 vs - 7PWei normalcase 19.796 ms/op 15.984 ms/op 1.24
altair processBlock - 250000 vs - 7PWei normalcase hashState 24.390 ms/op 26.783 ms/op 0.91
altair processBlock - 250000 vs - 7PWei worstcase 50.253 ms/op 46.406 ms/op 1.08
altair processBlock - 250000 vs - 7PWei worstcase hashState 71.111 ms/op 68.634 ms/op 1.04
phase0 processBlock - 250000 vs - 7PWei normalcase 2.0147 ms/op 2.0562 ms/op 0.98
phase0 processBlock - 250000 vs - 7PWei worstcase 27.883 ms/op 27.847 ms/op 1.00
altair processEth1Data - 250000 vs - 7PWei normalcase 462.33 us/op 442.02 us/op 1.05
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 7.1420 us/op 6.7640 us/op 1.06
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 20.233 us/op 19.220 us/op 1.05
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 8.5760 us/op 8.1070 us/op 1.06
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 6.5040 us/op 6.2160 us/op 1.05
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 75.795 us/op 73.879 us/op 1.03
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 648.73 us/op 610.25 us/op 1.06
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 929.80 us/op 903.01 us/op 1.03
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 847.21 us/op 894.76 us/op 0.95
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.2841 ms/op 2.2001 ms/op 1.04
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.5223 ms/op 1.4588 ms/op 1.04
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 3.7638 ms/op 3.8785 ms/op 0.97
Tree 40 250000 create 303.82 ms/op 288.25 ms/op 1.05
Tree 40 250000 get(125000) 180.84 ns/op 174.65 ns/op 1.04
Tree 40 250000 set(125000) 911.40 ns/op 862.14 ns/op 1.06
Tree 40 250000 toArray() 16.754 ms/op 17.835 ms/op 0.94
Tree 40 250000 iterate all - toArray() + loop 16.962 ms/op 17.679 ms/op 0.96
Tree 40 250000 iterate all - get(i) 66.330 ms/op 65.946 ms/op 1.01
MutableVector 250000 create 10.681 ms/op 9.9927 ms/op 1.07
MutableVector 250000 get(125000) 6.3320 ns/op 6.2920 ns/op 1.01
MutableVector 250000 set(125000) 257.29 ns/op 257.96 ns/op 1.00
MutableVector 250000 toArray() 2.6639 ms/op 2.7631 ms/op 0.96
MutableVector 250000 iterate all - toArray() + loop 2.7880 ms/op 3.0520 ms/op 0.91
MutableVector 250000 iterate all - get(i) 1.4566 ms/op 1.4954 ms/op 0.97
Array 250000 create 2.4854 ms/op 2.4647 ms/op 1.01
Array 250000 clone - spread 1.1254 ms/op 1.0899 ms/op 1.03
Array 250000 get(125000) 0.51400 ns/op 0.53200 ns/op 0.97
Array 250000 set(125000) 0.59700 ns/op 0.61000 ns/op 0.98
Array 250000 iterate all - loop 93.203 us/op 81.164 us/op 1.15
effectiveBalanceIncrements clone Uint8Array 300000 22.973 us/op 25.428 us/op 0.90
effectiveBalanceIncrements clone MutableVector 300000 321.00 ns/op 332.00 ns/op 0.97
effectiveBalanceIncrements rw all Uint8Array 300000 167.86 us/op 168.45 us/op 1.00
effectiveBalanceIncrements rw all MutableVector 300000 75.800 ms/op 80.141 ms/op 0.95
phase0 afterProcessEpoch - 250000 vs - 7PWei 108.34 ms/op 115.75 ms/op 0.94
phase0 beforeProcessEpoch - 250000 vs - 7PWei 34.130 ms/op 41.781 ms/op 0.82
altair processEpoch - mainnet_e81889 317.63 ms/op 327.14 ms/op 0.97
mainnet_e81889 - altair beforeProcessEpoch 60.701 ms/op 66.222 ms/op 0.92
mainnet_e81889 - altair processJustificationAndFinalization 16.190 us/op 17.694 us/op 0.91
mainnet_e81889 - altair processInactivityUpdates 5.3448 ms/op 5.3510 ms/op 1.00
mainnet_e81889 - altair processRewardsAndPenalties 66.374 ms/op 64.512 ms/op 1.03
mainnet_e81889 - altair processRegistryUpdates 2.8240 us/op 2.6900 us/op 1.05
mainnet_e81889 - altair processSlashings 509.00 ns/op 435.00 ns/op 1.17
mainnet_e81889 - altair processEth1DataReset 566.00 ns/op 467.00 ns/op 1.21
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.2270 ms/op 1.2497 ms/op 0.98
mainnet_e81889 - altair processSlashingsReset 4.3620 us/op 5.1750 us/op 0.84
mainnet_e81889 - altair processRandaoMixesReset 4.3610 us/op 4.3590 us/op 1.00
mainnet_e81889 - altair processHistoricalRootsUpdate 635.00 ns/op 551.00 ns/op 1.15
mainnet_e81889 - altair processParticipationFlagUpdates 2.8590 us/op 2.4600 us/op 1.16
mainnet_e81889 - altair processSyncCommitteeUpdates 611.00 ns/op 459.00 ns/op 1.33
mainnet_e81889 - altair afterProcessEpoch 121.59 ms/op 118.05 ms/op 1.03
phase0 processEpoch - mainnet_e58758 322.77 ms/op 324.45 ms/op 0.99
mainnet_e58758 - phase0 beforeProcessEpoch 122.78 ms/op 121.78 ms/op 1.01
mainnet_e58758 - phase0 processJustificationAndFinalization 17.545 us/op 17.256 us/op 1.02
mainnet_e58758 - phase0 processRewardsAndPenalties 55.174 ms/op 57.192 ms/op 0.96
mainnet_e58758 - phase0 processRegistryUpdates 8.0080 us/op 8.9660 us/op 0.89
mainnet_e58758 - phase0 processSlashings 565.00 ns/op 517.00 ns/op 1.09
mainnet_e58758 - phase0 processEth1DataReset 643.00 ns/op 519.00 ns/op 1.24
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 974.43 us/op 1.0598 ms/op 0.92
mainnet_e58758 - phase0 processSlashingsReset 3.0240 us/op 3.2620 us/op 0.93
mainnet_e58758 - phase0 processRandaoMixesReset 4.3610 us/op 6.3000 us/op 0.69
mainnet_e58758 - phase0 processHistoricalRootsUpdate 636.00 ns/op 853.00 ns/op 0.75
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.8550 us/op 5.0590 us/op 0.76
mainnet_e58758 - phase0 afterProcessEpoch 98.081 ms/op 96.933 ms/op 1.01
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.2766 ms/op 1.2387 ms/op 1.03
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.4866 ms/op 1.6224 ms/op 0.92
altair processInactivityUpdates - 250000 normalcase 23.157 ms/op 22.771 ms/op 1.02
altair processInactivityUpdates - 250000 worstcase 25.485 ms/op 27.318 ms/op 0.93
phase0 processRegistryUpdates - 250000 normalcase 6.2930 us/op 6.9710 us/op 0.90
phase0 processRegistryUpdates - 250000 badcase_full_deposits 273.42 us/op 270.46 us/op 1.01
phase0 processRegistryUpdates - 250000 worstcase 0.5 113.48 ms/op 110.48 ms/op 1.03
altair processRewardsAndPenalties - 250000 normalcase 47.062 ms/op 64.784 ms/op 0.73
altair processRewardsAndPenalties - 250000 worstcase 55.804 ms/op 70.075 ms/op 0.80
phase0 getAttestationDeltas - 250000 normalcase 6.6152 ms/op 6.6577 ms/op 0.99
phase0 getAttestationDeltas - 250000 worstcase 6.9801 ms/op 6.5876 ms/op 1.06
phase0 processSlashings - 250000 worstcase 3.7060 ms/op 3.3859 ms/op 1.09
altair processSyncCommitteeUpdates - 250000 170.41 ms/op 177.23 ms/op 0.96
BeaconState.hashTreeRoot - No change 272.00 ns/op 369.00 ns/op 0.74
BeaconState.hashTreeRoot - 1 full validator 49.952 us/op 50.937 us/op 0.98
BeaconState.hashTreeRoot - 32 full validator 504.39 us/op 549.17 us/op 0.92
BeaconState.hashTreeRoot - 512 full validator 5.0650 ms/op 5.3780 ms/op 0.94
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 62.962 us/op 60.712 us/op 1.04
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 895.00 us/op 881.17 us/op 1.02
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 11.726 ms/op 11.324 ms/op 1.04
BeaconState.hashTreeRoot - 1 balances 49.421 us/op 52.552 us/op 0.94
BeaconState.hashTreeRoot - 32 balances 430.95 us/op 450.77 us/op 0.96
BeaconState.hashTreeRoot - 512 balances 4.3067 ms/op 4.4361 ms/op 0.97
BeaconState.hashTreeRoot - 250000 balances 74.192 ms/op 70.658 ms/op 1.05
aggregationBits - 2048 els - zipIndexesInBitList 16.518 us/op 15.547 us/op 1.06
regular array get 100000 times 33.008 us/op 34.737 us/op 0.95
wrappedArray get 100000 times 32.981 us/op 32.758 us/op 1.01
arrayWithProxy get 100000 times 16.104 ms/op 15.679 ms/op 1.03
ssz.Root.equals 559.00 ns/op 545.00 ns/op 1.03
byteArrayEquals 550.00 ns/op 545.00 ns/op 1.01
shuffle list - 16384 els 6.8293 ms/op 6.7732 ms/op 1.01
shuffle list - 250000 els 100.48 ms/op 99.524 ms/op 1.01
processSlot - 1 slots 8.9590 us/op 8.8300 us/op 1.01
processSlot - 32 slots 1.3478 ms/op 1.3800 ms/op 0.98
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 33.518 ms/op 37.139 ms/op 0.90
getCommitteeAssignments - req 1 vs - 250000 vc 2.7785 ms/op 3.0980 ms/op 0.90
getCommitteeAssignments - req 100 vs - 250000 vc 3.9143 ms/op 4.3272 ms/op 0.90
getCommitteeAssignments - req 1000 vs - 250000 vc 4.2585 ms/op 4.5509 ms/op 0.94
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.1800 ns/op 4.7500 ns/op 0.88
state getBlockRootAtSlot - 250000 vs - 7PWei 863.49 ns/op 583.06 ns/op 1.48
computeProposers - vc 250000 10.503 ms/op 11.096 ms/op 0.95
computeEpochShuffling - vc 250000 99.608 ms/op 102.74 ms/op 0.97
getNextSyncCommittee - vc 250000 172.71 ms/op 179.00 ms/op 0.96
computeSigningRoot for AttestationData 13.327 us/op 13.860 us/op 0.96
hash AttestationData serialized data then Buffer.toString(base64) 2.3897 us/op 2.4531 us/op 0.97
toHexString serialized data 1.0702 us/op 1.0729 us/op 1.00
Buffer.toString(base64) 315.40 ns/op 326.00 ns/op 0.97

by benchmarkbot/action

@nflaig nflaig force-pushed the nflaig/apply-min-max-key-filtering branch from f1aaade to 3ed8d04 Compare April 29, 2023 20:18
@nflaig nflaig mentioned this pull request Apr 30, 2023
Closed
@nflaig nflaig changed the title fix(validator): filter by min-max key when loading pubkeys from db fix(slashing-protection): filter by min-max key when loading pubkeys from db May 2, 2023
@nflaig nflaig changed the title fix(slashing-protection): filter by min-max key when loading pubkeys from db fix(slashing-protection): filter by bucket when loading pubkeys from db May 3, 2023
@nflaig nflaig changed the title fix(slashing-protection): filter by bucket when loading pubkeys from db fix(slashing-protection): filter by db bucket when loading pubkeys May 3, 2023
@nflaig
Copy link
Member Author

nflaig commented May 4, 2023

Just to reconfirm, exporting a 200MB slashing protection file works on a VC with 2k keys, would consider the OOM issue fixed.

@dapplion dapplion merged commit 227d436 into unstable May 4, 2023
@dapplion dapplion deleted the nflaig/apply-min-max-key-filtering branch May 4, 2023 23:36
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.9.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dapplion dapplion dapplion approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

slashing protection exports goes OOM while exporting 500 validators kiln data
3 participants
@nflaig @wemeetagain @dapplion