Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use finalized state id when no checkpoint provided in checkpoint sync #5313

Merged
merged 2 commits into from
Mar 29, 2023
Merged

Use finalized state id when no checkpoint provided in checkpoint sync #5313

merged 2 commits into from
Mar 29, 2023

Conversation

g11tech
Copy link
Contributor

@g11tech g11tech commented Mar 28, 2023

When no checkpoint was provided, we used to explicitly fetch finalized checkpoint which caused issues on checkpointz as it would not cache states for those checkpoints whose root does not perfectly lie on epoch boundaries

This PR fixes that by just using finalized tag to get the state which checkpointz serves and duly updates with the last perfect checkpoint it has.

See: dappnode/DAppNodePackage-Lodestar-Prater#58

Closes #4524

@g11tech g11tech requested a review from a team as a code owner March 28, 2023 09:08
@g11tech g11tech enabled auto-merge (squash) March 28, 2023 09:10
@github-actions
Copy link
Contributor

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 14253ca Previous: f136f95 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 1.0704 ms/op 507.86 us/op 2.11
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 52.807 us/op 46.800 us/op 1.13
BLS verify - blst-native 1.2533 ms/op 1.2548 ms/op 1.00
BLS verifyMultipleSignatures 3 - blst-native 2.5448 ms/op 2.5153 ms/op 1.01
BLS verifyMultipleSignatures 8 - blst-native 5.4415 ms/op 5.4746 ms/op 0.99
BLS verifyMultipleSignatures 32 - blst-native 19.686 ms/op 19.677 ms/op 1.00
BLS aggregatePubkeys 32 - blst-native 28.213 us/op 26.229 us/op 1.08
BLS aggregatePubkeys 128 - blst-native 105.66 us/op 102.54 us/op 1.03
getAttestationsForBlock 66.944 ms/op 59.619 ms/op 1.12
isKnown best case - 1 super set check 273.00 ns/op 278.00 ns/op 0.98
isKnown normal case - 2 super set checks 264.00 ns/op 275.00 ns/op 0.96
isKnown worse case - 16 super set checks 260.00 ns/op 272.00 ns/op 0.96
CheckpointStateCache - add get delete 5.9190 us/op 5.4330 us/op 1.09
validate gossip signedAggregateAndProof - struct 2.8357 ms/op 2.8385 ms/op 1.00
validate gossip attestation - struct 1.3486 ms/op 1.3553 ms/op 1.00
pickEth1Vote - no votes 1.3478 ms/op 1.3654 ms/op 0.99
pickEth1Vote - max votes 12.410 ms/op 9.9112 ms/op 1.25
pickEth1Vote - Eth1Data hashTreeRoot value x2048 9.1930 ms/op 9.3754 ms/op 0.98
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 14.652 ms/op 15.362 ms/op 0.95
pickEth1Vote - Eth1Data fastSerialize value x2048 706.28 us/op 697.05 us/op 1.01
pickEth1Vote - Eth1Data fastSerialize tree x2048 6.8582 ms/op 6.8236 ms/op 1.01
bytes32 toHexString 592.00 ns/op 562.00 ns/op 1.05
bytes32 Buffer.toString(hex) 387.00 ns/op 365.00 ns/op 1.06
bytes32 Buffer.toString(hex) from Uint8Array 622.00 ns/op 619.00 ns/op 1.00
bytes32 Buffer.toString(hex) + 0x 408.00 ns/op 390.00 ns/op 1.05
Object access 1 prop 0.18500 ns/op 0.17700 ns/op 1.05
Map access 1 prop 0.17000 ns/op 0.16400 ns/op 1.04
Object get x1000 7.3450 ns/op 6.7900 ns/op 1.08
Map get x1000 0.75400 ns/op 0.64100 ns/op 1.18
Object set x1000 71.754 ns/op 58.132 ns/op 1.23
Map set x1000 55.244 ns/op 47.895 ns/op 1.15
Return object 10000 times 0.25890 ns/op 0.24870 ns/op 1.04
Throw Error 10000 times 4.2960 us/op 4.3884 us/op 0.98
fastMsgIdFn sha256 / 200 bytes 3.5800 us/op 3.6400 us/op 0.98
fastMsgIdFn h32 xxhash / 200 bytes 311.00 ns/op 321.00 ns/op 0.97
fastMsgIdFn h64 xxhash / 200 bytes 456.00 ns/op 469.00 ns/op 0.97
fastMsgIdFn sha256 / 1000 bytes 11.972 us/op 11.930 us/op 1.00
fastMsgIdFn h32 xxhash / 1000 bytes 463.00 ns/op 450.00 ns/op 1.03
fastMsgIdFn h64 xxhash / 1000 bytes 551.00 ns/op 561.00 ns/op 0.98
fastMsgIdFn sha256 / 10000 bytes 106.70 us/op 108.13 us/op 0.99
fastMsgIdFn h32 xxhash / 10000 bytes 2.0700 us/op 2.0650 us/op 1.00
fastMsgIdFn h64 xxhash / 10000 bytes 1.5130 us/op 1.4150 us/op 1.07
enrSubnets - fastDeserialize 64 bits 1.6270 us/op 1.8190 us/op 0.89
enrSubnets - ssz BitVector 64 bits 617.00 ns/op 640.00 ns/op 0.96
enrSubnets - fastDeserialize 4 bits 204.00 ns/op 216.00 ns/op 0.94
enrSubnets - ssz BitVector 4 bits 524.00 ns/op 618.00 ns/op 0.85
prioritizePeers score -10:0 att 32-0.1 sync 2-0 111.24 us/op 118.53 us/op 0.94
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 158.38 us/op 165.72 us/op 0.96
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 187.02 us/op 195.57 us/op 0.96
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 331.66 us/op 329.43 us/op 1.01
prioritizePeers score 0:0 att 64-1 sync 4-1 419.73 us/op 372.15 us/op 1.13
array of 16000 items push then shift 1.6970 us/op 1.6471 us/op 1.03
LinkedList of 16000 items push then shift 9.0900 ns/op 8.7100 ns/op 1.04
array of 16000 items push then pop 117.50 ns/op 88.887 ns/op 1.32
LinkedList of 16000 items push then pop 8.9230 ns/op 8.6010 ns/op 1.04
array of 24000 items push then shift 2.4063 us/op 2.3687 us/op 1.02
LinkedList of 24000 items push then shift 9.1740 ns/op 8.8520 ns/op 1.04
array of 24000 items push then pop 85.065 ns/op 77.180 ns/op 1.10
LinkedList of 24000 items push then pop 8.7800 ns/op 8.5020 ns/op 1.03
intersect bitArray bitLen 8 13.560 ns/op 13.201 ns/op 1.03
intersect array and set length 8 83.907 ns/op 78.115 ns/op 1.07
intersect bitArray bitLen 128 44.818 ns/op 43.977 ns/op 1.02
intersect array and set length 128 1.1681 us/op 1.0554 us/op 1.11
Buffer.concat 32 items 2.7490 us/op 2.7220 us/op 1.01
Uint8Array.set 32 items 2.3660 us/op 2.7860 us/op 0.85
pass gossip attestations to forkchoice per slot 3.9474 ms/op 2.8238 ms/op 1.40
computeDeltas 2.9124 ms/op 2.9339 ms/op 0.99
computeProposerBoostScoreFromBalances 1.8178 ms/op 1.8132 ms/op 1.00
altair processAttestation - 250000 vs - 7PWei normalcase 2.9830 ms/op 2.2121 ms/op 1.35
altair processAttestation - 250000 vs - 7PWei worstcase 3.7684 ms/op 3.4600 ms/op 1.09
altair processAttestation - setStatus - 1/6 committees join 153.14 us/op 141.73 us/op 1.08
altair processAttestation - setStatus - 1/3 committees join 278.90 us/op 281.20 us/op 0.99
altair processAttestation - setStatus - 1/2 committees join 382.18 us/op 380.68 us/op 1.00
altair processAttestation - setStatus - 2/3 committees join 477.41 us/op 473.85 us/op 1.01
altair processAttestation - setStatus - 4/5 committees join 669.37 us/op 672.78 us/op 0.99
altair processAttestation - setStatus - 100% committees join 779.36 us/op 763.38 us/op 1.02
altair processBlock - 250000 vs - 7PWei normalcase 17.722 ms/op 16.957 ms/op 1.05
altair processBlock - 250000 vs - 7PWei normalcase hashState 27.577 ms/op 24.507 ms/op 1.13
altair processBlock - 250000 vs - 7PWei worstcase 49.110 ms/op 53.937 ms/op 0.91
altair processBlock - 250000 vs - 7PWei worstcase hashState 77.930 ms/op 65.343 ms/op 1.19
phase0 processBlock - 250000 vs - 7PWei normalcase 2.3032 ms/op 2.2108 ms/op 1.04
phase0 processBlock - 250000 vs - 7PWei worstcase 32.260 ms/op 29.215 ms/op 1.10
altair processEth1Data - 250000 vs - 7PWei normalcase 668.97 us/op 507.57 us/op 1.32
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 10.883 us/op 7.4740 us/op 1.46
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 32.381 us/op 22.834 us/op 1.42
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 15.398 us/op 8.5150 us/op 1.81
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 12.158 us/op 7.2040 us/op 1.69
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 129.69 us/op 94.092 us/op 1.38
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 1.1795 ms/op 608.37 us/op 1.94
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 1.4833 ms/op 910.45 us/op 1.63
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 1.3868 ms/op 849.51 us/op 1.63
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 4.3049 ms/op 2.2744 ms/op 1.89
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 2.8078 ms/op 1.4568 ms/op 1.93
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 10.291 ms/op 3.7981 ms/op 2.71
Tree 40 250000 create 930.18 ms/op 310.03 ms/op 3.00
Tree 40 250000 get(125000) 305.92 ns/op 180.20 ns/op 1.70
Tree 40 250000 set(125000) 3.6941 us/op 961.85 ns/op 3.84
Tree 40 250000 toArray() 36.719 ms/op 17.853 ms/op 2.06
Tree 40 250000 iterate all - toArray() + loop 40.456 ms/op 16.422 ms/op 2.46
Tree 40 250000 iterate all - get(i) 126.25 ms/op 66.003 ms/op 1.91
MutableVector 250000 create 17.409 ms/op 9.6460 ms/op 1.80
MutableVector 250000 get(125000) 10.884 ns/op 6.6870 ns/op 1.63
MutableVector 250000 set(125000) 893.57 ns/op 258.14 ns/op 3.46
MutableVector 250000 toArray() 6.5896 ms/op 2.8799 ms/op 2.29
MutableVector 250000 iterate all - toArray() + loop 5.5366 ms/op 2.9318 ms/op 1.89
MutableVector 250000 iterate all - get(i) 2.6606 ms/op 1.5148 ms/op 1.76
Array 250000 create 4.7038 ms/op 2.5274 ms/op 1.86
Array 250000 clone - spread 1.7288 ms/op 1.1485 ms/op 1.51
Array 250000 get(125000) 1.6330 ns/op 0.56400 ns/op 2.90
Array 250000 set(125000) 1.6360 ns/op 0.63100 ns/op 2.59
Array 250000 iterate all - loop 94.716 us/op 107.34 us/op 0.88
effectiveBalanceIncrements clone Uint8Array 300000 70.242 us/op 26.095 us/op 2.69
effectiveBalanceIncrements clone MutableVector 300000 1.2980 us/op 359.00 ns/op 3.62
effectiveBalanceIncrements rw all Uint8Array 300000 187.56 us/op 163.05 us/op 1.15
effectiveBalanceIncrements rw all MutableVector 300000 211.82 ms/op 79.113 ms/op 2.68
phase0 afterProcessEpoch - 250000 vs - 7PWei 144.62 ms/op 111.95 ms/op 1.29
phase0 beforeProcessEpoch - 250000 vs - 7PWei 60.016 ms/op 39.252 ms/op 1.53
altair processEpoch - mainnet_e81889 514.33 ms/op 330.06 ms/op 1.56
mainnet_e81889 - altair beforeProcessEpoch 144.31 ms/op 61.891 ms/op 2.33
mainnet_e81889 - altair processJustificationAndFinalization 36.153 us/op 17.254 us/op 2.10
mainnet_e81889 - altair processInactivityUpdates 14.314 ms/op 5.5043 ms/op 2.60
mainnet_e81889 - altair processRewardsAndPenalties 83.929 ms/op 67.450 ms/op 1.24
mainnet_e81889 - altair processRegistryUpdates 7.6850 us/op 2.5470 us/op 3.02
mainnet_e81889 - altair processSlashings 1.7930 us/op 474.00 ns/op 3.78
mainnet_e81889 - altair processEth1DataReset 2.0110 us/op 516.00 ns/op 3.90
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.0108 ms/op 1.2259 ms/op 1.64
mainnet_e81889 - altair processSlashingsReset 8.0300 us/op 4.5930 us/op 1.75
mainnet_e81889 - altair processRandaoMixesReset 11.279 us/op 4.5480 us/op 2.48
mainnet_e81889 - altair processHistoricalRootsUpdate 1.4680 us/op 603.00 ns/op 2.43
mainnet_e81889 - altair processParticipationFlagUpdates 3.7000 us/op 2.3600 us/op 1.57
mainnet_e81889 - altair processSyncCommitteeUpdates 1.2180 us/op 702.00 ns/op 1.74
mainnet_e81889 - altair afterProcessEpoch 121.76 ms/op 125.03 ms/op 0.97
phase0 processEpoch - mainnet_e58758 370.32 ms/op 335.24 ms/op 1.10
mainnet_e58758 - phase0 beforeProcessEpoch 168.70 ms/op 121.17 ms/op 1.39
mainnet_e58758 - phase0 processJustificationAndFinalization 18.191 us/op 17.726 us/op 1.03
mainnet_e58758 - phase0 processRewardsAndPenalties 60.592 ms/op 53.322 ms/op 1.14
mainnet_e58758 - phase0 processRegistryUpdates 9.5180 us/op 7.9040 us/op 1.20
mainnet_e58758 - phase0 processSlashings 553.00 ns/op 482.00 ns/op 1.15
mainnet_e58758 - phase0 processEth1DataReset 1.1320 us/op 511.00 ns/op 2.22
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.7543 ms/op 937.04 us/op 1.87
mainnet_e58758 - phase0 processSlashingsReset 6.6420 us/op 3.9170 us/op 1.70
mainnet_e58758 - phase0 processRandaoMixesReset 8.1600 us/op 4.7540 us/op 1.72
mainnet_e58758 - phase0 processHistoricalRootsUpdate 1.2560 us/op 559.00 ns/op 2.25
mainnet_e58758 - phase0 processParticipationRecordUpdates 9.7540 us/op 3.8540 us/op 2.53
mainnet_e58758 - phase0 afterProcessEpoch 135.76 ms/op 96.234 ms/op 1.41
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.9934 ms/op 1.2351 ms/op 1.61
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 3.6880 ms/op 1.4568 ms/op 2.53
altair processInactivityUpdates - 250000 normalcase 38.380 ms/op 20.797 ms/op 1.85
altair processInactivityUpdates - 250000 worstcase 36.020 ms/op 22.975 ms/op 1.57
phase0 processRegistryUpdates - 250000 normalcase 13.073 us/op 5.9670 us/op 2.19
phase0 processRegistryUpdates - 250000 badcase_full_deposits 455.86 us/op 239.35 us/op 1.90
phase0 processRegistryUpdates - 250000 worstcase 0.5 195.81 ms/op 122.94 ms/op 1.59
altair processRewardsAndPenalties - 250000 normalcase 68.578 ms/op 57.816 ms/op 1.19
altair processRewardsAndPenalties - 250000 worstcase 83.155 ms/op 59.806 ms/op 1.39
phase0 getAttestationDeltas - 250000 normalcase 12.142 ms/op 6.3414 ms/op 1.91
phase0 getAttestationDeltas - 250000 worstcase 12.215 ms/op 6.3476 ms/op 1.92
phase0 processSlashings - 250000 worstcase 6.2381 ms/op 3.5908 ms/op 1.74
altair processSyncCommitteeUpdates - 250000 295.88 ms/op 174.09 ms/op 1.70
BeaconState.hashTreeRoot - No change 354.00 ns/op 261.00 ns/op 1.36
BeaconState.hashTreeRoot - 1 full validator 74.535 us/op 48.127 us/op 1.55
BeaconState.hashTreeRoot - 32 full validator 718.61 us/op 510.39 us/op 1.41
BeaconState.hashTreeRoot - 512 full validator 9.7149 ms/op 5.1838 ms/op 1.87
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 90.405 us/op 63.735 us/op 1.42
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.3021 ms/op 854.24 us/op 1.52
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 19.032 ms/op 11.317 ms/op 1.68
BeaconState.hashTreeRoot - 1 balances 65.740 us/op 49.155 us/op 1.34
BeaconState.hashTreeRoot - 32 balances 638.88 us/op 430.87 us/op 1.48
BeaconState.hashTreeRoot - 512 balances 7.0109 ms/op 4.2798 ms/op 1.64
BeaconState.hashTreeRoot - 250000 balances 114.50 ms/op 74.429 ms/op 1.54
aggregationBits - 2048 els - zipIndexesInBitList 39.402 us/op 16.074 us/op 2.45
regular array get 100000 times 93.718 us/op 32.082 us/op 2.92
wrappedArray get 100000 times 94.067 us/op 32.068 us/op 2.93
arrayWithProxy get 100000 times 29.259 ms/op 15.602 ms/op 1.88
ssz.Root.equals 1.1460 us/op 534.00 ns/op 2.15
byteArrayEquals 1.1240 us/op 522.00 ns/op 2.15
shuffle list - 16384 els 10.371 ms/op 6.8084 ms/op 1.52
shuffle list - 250000 els 148.65 ms/op 99.303 ms/op 1.50
processSlot - 1 slots 16.014 us/op 8.5220 us/op 1.88
processSlot - 32 slots 1.9845 ms/op 1.3282 ms/op 1.49
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 40.967 ms/op 34.251 ms/op 1.20
getCommitteeAssignments - req 1 vs - 250000 vc 3.5673 ms/op 2.8589 ms/op 1.25
getCommitteeAssignments - req 100 vs - 250000 vc 5.0757 ms/op 4.0572 ms/op 1.25
getCommitteeAssignments - req 1000 vs - 250000 vc 5.0229 ms/op 4.3715 ms/op 1.15
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.6800 ns/op 4.6100 ns/op 1.23
state getBlockRootAtSlot - 250000 vs - 7PWei 858.54 ns/op 785.78 ns/op 1.09
computeProposers - vc 250000 12.429 ms/op 10.387 ms/op 1.20
computeEpochShuffling - vc 250000 122.38 ms/op 100.96 ms/op 1.21
getNextSyncCommittee - vc 250000 220.74 ms/op 174.52 ms/op 1.26

by benchmarkbot/action

@g11tech g11tech merged commit f3df9f8 into unstable Mar 29, 2023
@g11tech g11tech deleted the g11tech/checkpointz-fix branch March 29, 2023 03:03
philknows pushed a commit that referenced this pull request Mar 29, 2023
@g11tech @philknows
Use finalized state id when no checkpoint provided in checkpoint sync (…
baa95e4 
…#5313)

* Use finalized state id when no checkpoint provided in checkpoint sync

* add stateid to log
@philknows philknows mentioned this pull request Mar 29, 2023
Merged
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.7.1 🎉

@wemeetagain
Copy link
Member

🎉 This PR is included in v1.8.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dapplion dapplion dapplion approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

When using --checkpointsync if the flag isn't at the head (see image) the sync download fails
3 participants
@g11tech @wemeetagain @dapplion