Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: add capella support to sim multi-fork tests #5489

Merged
merged 18 commits into from
May 18, 2023
Merged

test: add capella support to sim multi-fork tests #5489

merged 18 commits into from
May 18, 2023

Conversation

nazarhussain
Copy link
Contributor

Motivation

Keep the sim tests up-to-date.

Description

Add capella fork support to multi-fork sim tests.

Closes #5460

Steps to test or reproduce

  • Run all sim tests

@nazarhussain nazarhussain self-assigned this May 12, 2023
@github-actions
Copy link
Contributor

github-actions bot commented May 12, 2023 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: c11a3ef Previous: 1602f67 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 840.75 us/op 630.64 us/op 1.33
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 46.126 us/op 46.936 us/op 0.98
BLS verify - blst-native 1.2279 ms/op 1.2233 ms/op 1.00
BLS verifyMultipleSignatures 3 - blst-native 2.4918 ms/op 2.4936 ms/op 1.00
BLS verifyMultipleSignatures 8 - blst-native 5.3529 ms/op 5.3491 ms/op 1.00
BLS verifyMultipleSignatures 32 - blst-native 19.362 ms/op 19.367 ms/op 1.00
BLS aggregatePubkeys 32 - blst-native 26.047 us/op 25.999 us/op 1.00
BLS aggregatePubkeys 128 - blst-native 100.80 us/op 101.47 us/op 0.99
getAttestationsForBlock 52.033 ms/op 64.347 ms/op 0.81
isKnown best case - 1 super set check 260.00 ns/op 276.00 ns/op 0.94
isKnown normal case - 2 super set checks 254.00 ns/op 260.00 ns/op 0.98
isKnown worse case - 16 super set checks 253.00 ns/op 259.00 ns/op 0.98
CheckpointStateCache - add get delete 5.5770 us/op 5.4730 us/op 1.02
validate gossip signedAggregateAndProof - struct 2.8122 ms/op 2.8103 ms/op 1.00
validate gossip attestation - struct 1.3384 ms/op 1.3659 ms/op 0.98
pickEth1Vote - no votes 1.3774 ms/op 1.4015 ms/op 0.98
pickEth1Vote - max votes 8.9763 ms/op 11.239 ms/op 0.80
pickEth1Vote - Eth1Data hashTreeRoot value x2048 9.3409 ms/op 9.6553 ms/op 0.97
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 15.631 ms/op 15.691 ms/op 1.00
pickEth1Vote - Eth1Data fastSerialize value x2048 718.10 us/op 670.59 us/op 1.07
pickEth1Vote - Eth1Data fastSerialize tree x2048 5.5295 ms/op 6.5932 ms/op 0.84
bytes32 toHexString 492.00 ns/op 493.00 ns/op 1.00
bytes32 Buffer.toString(hex) 347.00 ns/op 344.00 ns/op 1.01
bytes32 Buffer.toString(hex) from Uint8Array 528.00 ns/op 565.00 ns/op 0.93
bytes32 Buffer.toString(hex) + 0x 352.00 ns/op 360.00 ns/op 0.98
Object access 1 prop 0.17300 ns/op 0.17300 ns/op 1.00
Map access 1 prop 0.16300 ns/op 0.17700 ns/op 0.92
Object get x1000 6.5640 ns/op 7.6670 ns/op 0.86
Map get x1000 0.54200 ns/op 0.61500 ns/op 0.88
Object set x1000 52.829 ns/op 63.167 ns/op 0.84
Map set x1000 43.758 ns/op 44.455 ns/op 0.98
Return object 10000 times 0.23660 ns/op 0.23910 ns/op 0.99
Throw Error 10000 times 4.2269 us/op 4.2676 us/op 0.99
fastMsgIdFn sha256 / 200 bytes 3.4700 us/op 3.5620 us/op 0.97
fastMsgIdFn h32 xxhash / 200 bytes 283.00 ns/op 282.00 ns/op 1.00
fastMsgIdFn h64 xxhash / 200 bytes 407.00 ns/op 409.00 ns/op 1.00
fastMsgIdFn sha256 / 1000 bytes 11.704 us/op 11.712 us/op 1.00
fastMsgIdFn h32 xxhash / 1000 bytes 408.00 ns/op 422.00 ns/op 0.97
fastMsgIdFn h64 xxhash / 1000 bytes 471.00 ns/op 490.00 ns/op 0.96
fastMsgIdFn sha256 / 10000 bytes 104.07 us/op 105.39 us/op 0.99
fastMsgIdFn h32 xxhash / 10000 bytes 1.9310 us/op 1.9770 us/op 0.98
fastMsgIdFn h64 xxhash / 10000 bytes 1.3800 us/op 1.3970 us/op 0.99
enrSubnets - fastDeserialize 64 bits 1.2820 us/op 1.3240 us/op 0.97
enrSubnets - ssz BitVector 64 bits 530.00 ns/op 514.00 ns/op 1.03
enrSubnets - fastDeserialize 4 bits 174.00 ns/op 172.00 ns/op 1.01
enrSubnets - ssz BitVector 4 bits 523.00 ns/op 531.00 ns/op 0.98
prioritizePeers score -10:0 att 32-0.1 sync 2-0 115.10 us/op 110.73 us/op 1.04
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 148.51 us/op 137.95 us/op 1.08
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 183.50 us/op 189.51 us/op 0.97
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 345.22 us/op 337.41 us/op 1.02
prioritizePeers score 0:0 att 64-1 sync 4-1 406.46 us/op 413.75 us/op 0.98
array of 16000 items push then shift 1.6605 us/op 1.6905 us/op 0.98
LinkedList of 16000 items push then shift 8.9310 ns/op 9.3430 ns/op 0.96
array of 16000 items push then pop 111.03 ns/op 109.14 ns/op 1.02
LinkedList of 16000 items push then pop 8.8380 ns/op 9.0290 ns/op 0.98
array of 24000 items push then shift 2.2748 us/op 2.4070 us/op 0.95
LinkedList of 24000 items push then shift 9.0780 ns/op 9.5870 ns/op 0.95
array of 24000 items push then pop 87.262 ns/op 92.390 ns/op 0.94
LinkedList of 24000 items push then pop 8.8540 ns/op 9.2060 ns/op 0.96
intersect bitArray bitLen 8 13.793 ns/op 13.771 ns/op 1.00
intersect array and set length 8 85.890 ns/op 84.476 ns/op 1.02
intersect bitArray bitLen 128 45.140 ns/op 45.057 ns/op 1.00
intersect array and set length 128 1.2402 us/op 1.1600 us/op 1.07
Buffer.concat 32 items 3.1480 us/op 2.8830 us/op 1.09
Uint8Array.set 32 items 3.1120 us/op 2.3430 us/op 1.33
pass gossip attestations to forkchoice per slot 2.7686 ms/op 2.3937 ms/op 1.16
computeDeltas 2.9408 ms/op 3.0833 ms/op 0.95
computeProposerBoostScoreFromBalances 1.7790 ms/op 1.8290 ms/op 0.97
altair processAttestation - 250000 vs - 7PWei normalcase 2.3750 ms/op 2.9525 ms/op 0.80
altair processAttestation - 250000 vs - 7PWei worstcase 3.5113 ms/op 4.2302 ms/op 0.83
altair processAttestation - setStatus - 1/6 committees join 153.40 us/op 145.42 us/op 1.05
altair processAttestation - setStatus - 1/3 committees join 284.55 us/op 295.72 us/op 0.96
altair processAttestation - setStatus - 1/2 committees join 373.83 us/op 386.23 us/op 0.97
altair processAttestation - setStatus - 2/3 committees join 461.68 us/op 490.38 us/op 0.94
altair processAttestation - setStatus - 4/5 committees join 684.29 us/op 697.87 us/op 0.98
altair processAttestation - setStatus - 100% committees join 767.89 us/op 818.04 us/op 0.94
altair processBlock - 250000 vs - 7PWei normalcase 15.368 ms/op 20.112 ms/op 0.76
altair processBlock - 250000 vs - 7PWei normalcase hashState 25.208 ms/op 30.417 ms/op 0.83
altair processBlock - 250000 vs - 7PWei worstcase 47.421 ms/op 56.604 ms/op 0.84
altair processBlock - 250000 vs - 7PWei worstcase hashState 69.459 ms/op 74.745 ms/op 0.93
phase0 processBlock - 250000 vs - 7PWei normalcase 1.9946 ms/op 2.1121 ms/op 0.94
phase0 processBlock - 250000 vs - 7PWei worstcase 28.177 ms/op 31.217 ms/op 0.90
altair processEth1Data - 250000 vs - 7PWei normalcase 461.81 us/op 556.63 us/op 0.83
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 7.8930 us/op 9.4260 us/op 0.84
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 24.870 us/op 29.673 us/op 0.84
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 10.035 us/op 10.559 us/op 0.95
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 6.9400 us/op 7.9540 us/op 0.87
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 76.568 us/op 118.10 us/op 0.65
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 643.17 us/op 777.35 us/op 0.83
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 910.84 us/op 1.0613 ms/op 0.86
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 913.44 us/op 1.0182 ms/op 0.90
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.4140 ms/op 2.4917 ms/op 0.97
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.5761 ms/op 1.6340 ms/op 0.96
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 3.8533 ms/op 4.2136 ms/op 0.91
Tree 40 250000 create 351.39 ms/op 433.02 ms/op 0.81
Tree 40 250000 get(125000) 199.94 ns/op 194.75 ns/op 1.03
Tree 40 250000 set(125000) 1.0142 us/op 1.0200 us/op 0.99
Tree 40 250000 toArray() 20.974 ms/op 21.627 ms/op 0.97
Tree 40 250000 iterate all - toArray() + loop 20.868 ms/op 22.368 ms/op 0.93
Tree 40 250000 iterate all - get(i) 74.423 ms/op 74.799 ms/op 0.99
MutableVector 250000 create 11.476 ms/op 14.199 ms/op 0.81
MutableVector 250000 get(125000) 6.4160 ns/op 7.0750 ns/op 0.91
MutableVector 250000 set(125000) 272.92 ns/op 353.07 ns/op 0.77
MutableVector 250000 toArray() 2.9091 ms/op 3.8956 ms/op 0.75
MutableVector 250000 iterate all - toArray() + loop 3.0465 ms/op 3.5055 ms/op 0.87
MutableVector 250000 iterate all - get(i) 1.5273 ms/op 1.5692 ms/op 0.97
Array 250000 create 3.6449 ms/op 3.3929 ms/op 1.07
Array 250000 clone - spread 1.1608 ms/op 1.3117 ms/op 0.88
Array 250000 get(125000) 0.54800 ns/op 0.64400 ns/op 0.85
Array 250000 set(125000) 0.64000 ns/op 0.68900 ns/op 0.93
Array 250000 iterate all - loop 85.694 us/op 100.37 us/op 0.85
effectiveBalanceIncrements clone Uint8Array 300000 33.767 us/op 36.702 us/op 0.92
effectiveBalanceIncrements clone MutableVector 300000 337.00 ns/op 403.00 ns/op 0.84
effectiveBalanceIncrements rw all Uint8Array 300000 174.83 us/op 174.68 us/op 1.00
effectiveBalanceIncrements rw all MutableVector 300000 83.877 ms/op 95.702 ms/op 0.88
phase0 afterProcessEpoch - 250000 vs - 7PWei 116.99 ms/op 123.89 ms/op 0.94
phase0 beforeProcessEpoch - 250000 vs - 7PWei 37.236 ms/op 42.342 ms/op 0.88
altair processEpoch - mainnet_e81889 307.03 ms/op 371.28 ms/op 0.83
mainnet_e81889 - altair beforeProcessEpoch 51.877 ms/op 78.687 ms/op 0.66
mainnet_e81889 - altair processJustificationAndFinalization 15.500 us/op 24.831 us/op 0.62
mainnet_e81889 - altair processInactivityUpdates 5.7951 ms/op 6.4491 ms/op 0.90
mainnet_e81889 - altair processRewardsAndPenalties 67.512 ms/op 76.295 ms/op 0.88
mainnet_e81889 - altair processRegistryUpdates 2.2630 us/op 3.7760 us/op 0.60
mainnet_e81889 - altair processSlashings 458.00 ns/op 1.1220 us/op 0.41
mainnet_e81889 - altair processEth1DataReset 465.00 ns/op 1.2340 us/op 0.38
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.2377 ms/op 1.5343 ms/op 0.81
mainnet_e81889 - altair processSlashingsReset 3.7080 us/op 7.3000 us/op 0.51
mainnet_e81889 - altair processRandaoMixesReset 4.4640 us/op 8.4850 us/op 0.53
mainnet_e81889 - altair processHistoricalRootsUpdate 577.00 ns/op 1.7210 us/op 0.34
mainnet_e81889 - altair processParticipationFlagUpdates 2.6120 us/op 3.6740 us/op 0.71
mainnet_e81889 - altair processSyncCommitteeUpdates 539.00 ns/op 942.00 ns/op 0.57
mainnet_e81889 - altair afterProcessEpoch 128.69 ms/op 133.49 ms/op 0.96
phase0 processEpoch - mainnet_e58758 361.23 ms/op 454.26 ms/op 0.80
mainnet_e58758 - phase0 beforeProcessEpoch 127.32 ms/op 219.37 ms/op 0.58
mainnet_e58758 - phase0 processJustificationAndFinalization 19.247 us/op 30.516 us/op 0.63
mainnet_e58758 - phase0 processRewardsAndPenalties 60.897 ms/op 74.718 ms/op 0.82
mainnet_e58758 - phase0 processRegistryUpdates 8.9890 us/op 12.882 us/op 0.70
mainnet_e58758 - phase0 processSlashings 529.00 ns/op 1.0190 us/op 0.52
mainnet_e58758 - phase0 processEth1DataReset 473.00 ns/op 1.0980 us/op 0.43
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 3.0130 ms/op 1.3750 ms/op 2.19
mainnet_e58758 - phase0 processSlashingsReset 4.8200 us/op 7.0330 us/op 0.69
mainnet_e58758 - phase0 processRandaoMixesReset 5.1640 us/op 9.1860 us/op 0.56
mainnet_e58758 - phase0 processHistoricalRootsUpdate 589.00 ns/op 1.4510 us/op 0.41
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.6310 us/op 11.301 us/op 0.32
mainnet_e58758 - phase0 afterProcessEpoch 97.072 ms/op 115.31 ms/op 0.84
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.2201 ms/op 1.5469 ms/op 0.79
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.5793 ms/op 1.9287 ms/op 0.82
altair processInactivityUpdates - 250000 normalcase 19.551 ms/op 28.802 ms/op 0.68
altair processInactivityUpdates - 250000 worstcase 25.313 ms/op 30.741 ms/op 0.82
phase0 processRegistryUpdates - 250000 normalcase 7.0640 us/op 10.647 us/op 0.66
phase0 processRegistryUpdates - 250000 badcase_full_deposits 246.34 us/op 376.08 us/op 0.66
phase0 processRegistryUpdates - 250000 worstcase 0.5 118.81 ms/op 140.94 ms/op 0.84
altair processRewardsAndPenalties - 250000 normalcase 64.881 ms/op 78.214 ms/op 0.83
altair processRewardsAndPenalties - 250000 worstcase 68.608 ms/op 93.113 ms/op 0.74
phase0 getAttestationDeltas - 250000 normalcase 6.6310 ms/op 10.671 ms/op 0.62
phase0 getAttestationDeltas - 250000 worstcase 6.5587 ms/op 9.8250 ms/op 0.67
phase0 processSlashings - 250000 worstcase 3.5542 ms/op 4.3182 ms/op 0.82
altair processSyncCommitteeUpdates - 250000 175.84 ms/op 217.03 ms/op 0.81
BeaconState.hashTreeRoot - No change 267.00 ns/op 312.00 ns/op 0.86
BeaconState.hashTreeRoot - 1 full validator 52.116 us/op 63.766 us/op 0.82
BeaconState.hashTreeRoot - 32 full validator 545.60 us/op 596.72 us/op 0.91
BeaconState.hashTreeRoot - 512 full validator 5.2298 ms/op 6.5059 ms/op 0.80
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 62.701 us/op 68.710 us/op 0.91
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 869.50 us/op 984.33 us/op 0.88
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 11.042 ms/op 16.077 ms/op 0.69
BeaconState.hashTreeRoot - 1 balances 47.726 us/op 53.003 us/op 0.90
BeaconState.hashTreeRoot - 32 balances 414.50 us/op 534.63 us/op 0.78
BeaconState.hashTreeRoot - 512 balances 4.2740 ms/op 4.7708 ms/op 0.90
BeaconState.hashTreeRoot - 250000 balances 78.687 ms/op 84.047 ms/op 0.94
aggregationBits - 2048 els - zipIndexesInBitList 15.833 us/op 19.191 us/op 0.83
regular array get 100000 times 33.253 us/op 43.915 us/op 0.76
wrappedArray get 100000 times 33.042 us/op 34.673 us/op 0.95
arrayWithProxy get 100000 times 15.526 ms/op 17.291 ms/op 0.90
ssz.Root.equals 547.00 ns/op 582.00 ns/op 0.94
byteArrayEquals 547.00 ns/op 584.00 ns/op 0.94
shuffle list - 16384 els 6.8799 ms/op 7.1094 ms/op 0.97
shuffle list - 250000 els 100.63 ms/op 104.32 ms/op 0.96
processSlot - 1 slots 9.2450 us/op 13.365 us/op 0.69
processSlot - 32 slots 1.3283 ms/op 1.6532 ms/op 0.80
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 36.582 ms/op 42.212 ms/op 0.87
getCommitteeAssignments - req 1 vs - 250000 vc 2.9598 ms/op 2.9801 ms/op 0.99
getCommitteeAssignments - req 100 vs - 250000 vc 4.1565 ms/op 4.2669 ms/op 0.97
getCommitteeAssignments - req 1000 vs - 250000 vc 4.5315 ms/op 4.5523 ms/op 1.00
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.5300 ns/op 5.0100 ns/op 0.90
state getBlockRootAtSlot - 250000 vs - 7PWei 922.70 ns/op 1.1022 us/op 0.84
computeProposers - vc 250000 10.757 ms/op 10.568 ms/op 1.02
computeEpochShuffling - vc 250000 103.96 ms/op 107.26 ms/op 0.97
getNextSyncCommittee - vc 250000 181.12 ms/op 185.46 ms/op 0.98
computeSigningRoot for AttestationData 13.842 us/op 13.695 us/op 1.01
hash AttestationData serialized data then Buffer.toString(base64) 2.4906 us/op 2.4222 us/op 1.03
toHexString serialized data 1.0778 us/op 1.0524 us/op 1.02
Buffer.toString(base64) 321.38 ns/op 325.72 ns/op 0.99

by benchmarkbot/action

@dapplion
Copy link
Contributor

Looks good overall! To prevent this from happening again, could you implement a check such that:

  • look at all forks that are scheduled in mainnet
  • require all those forks to be touched by this test

@nazarhussain nazarhussain changed the title feat: add capella support to sim multi-fork tests test: add capella support to sim multi-fork tests May 15, 2023
@nazarhussain nazarhussain marked this pull request as ready for review May 16, 2023 23:29
@nazarhussain nazarhussain requested a review from a team as a code owner May 16, 2023 23:29
dapplion
dapplion reviewed May 17, 2023
View reviewed changes
packages/cli/test/sim/multi_fork.test.ts
await waitForSlot(env.clock.getLastSlotOfEpoch(bellatrixForkEpoch) + activePreset.SLOTS_PER_EPOCH / 2, env.nodes, {
silent: true,
let lastForkEpoch = 0;
for (const fork of env.forkConfig.forksAscendingEpochOrder) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a nice comment above this code explaining the purpose of this and motivation

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@dapplion dapplion merged commit 1939c37 into unstable May 18, 2023
@dapplion dapplion deleted the nh/5460-capella-support-sim-tests branch May 18, 2023 12:34
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.9.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dapplion dapplion dapplion approved these changes

Assignees

@nazarhussain nazarhussain

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add capella to the multi-fork sim tests
3 participants
@nazarhussain @dapplion @wemeetagain