Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Review metrics #4475

Merged
merged 3 commits into from
Sep 1, 2022
Merged

Review metrics #4475

merged 3 commits into from
Sep 1, 2022

Conversation

dapplion
Copy link
Contributor

Motivation

Misc clean-up of existing metrics

Description

  • Remove un-used metrics
  • Remove index label from validator monitor
  • Use histograms instead of avgMinMax
  • Use proper bucket sizes
  • Review metrics
  • Move fork-choice metrics to beacon-node package

@dapplion dapplion requested a review from a team as a code owner August 24, 2022 12:30
@dapplion dapplion added the scope-metrics All issues with regards to the exposed metrics. label Aug 24, 2022
@dapplion dapplion mentioned this pull request Aug 24, 2022
Closed
@github-actions
Copy link
Contributor

github-actions bot commented Aug 24, 2022 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 0e88a63 Previous: aa97981 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 1.9124 ms/op 2.2164 ms/op 0.86
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 77.865 us/op 72.669 us/op 1.07
BLS verify - blst-native 2.0935 ms/op 1.8553 ms/op 1.13
BLS verifyMultipleSignatures 3 - blst-native 4.1874 ms/op 3.8067 ms/op 1.10
BLS verifyMultipleSignatures 8 - blst-native 9.1727 ms/op 8.1912 ms/op 1.12
BLS verifyMultipleSignatures 32 - blst-native 34.102 ms/op 29.708 ms/op 1.15
BLS aggregatePubkeys 32 - blst-native 43.449 us/op 39.178 us/op 1.11
BLS aggregatePubkeys 128 - blst-native 172.02 us/op 152.96 us/op 1.12
getAttestationsForBlock 185.57 ms/op 170.30 ms/op 1.09
isKnown best case - 1 super set check 469.00 ns/op 440.00 ns/op 1.07
isKnown normal case - 2 super set checks 482.00 ns/op 431.00 ns/op 1.12
isKnown worse case - 16 super set checks 469.00 ns/op 440.00 ns/op 1.07
CheckpointStateCache - add get delete 9.5020 us/op 9.2940 us/op 1.02
validate gossip signedAggregateAndProof - struct 4.7370 ms/op 4.2678 ms/op 1.11
validate gossip attestation - struct 2.2492 ms/op 2.0282 ms/op 1.11
pickEth1Vote - no votes 2.4611 ms/op 2.1832 ms/op 1.13
pickEth1Vote - max votes 19.953 ms/op 20.032 ms/op 1.00
pickEth1Vote - Eth1Data hashTreeRoot value x2048 11.557 ms/op 11.547 ms/op 1.00
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 22.803 ms/op 21.593 ms/op 1.06
pickEth1Vote - Eth1Data fastSerialize value x2048 1.8268 ms/op 1.5873 ms/op 1.15
pickEth1Vote - Eth1Data fastSerialize tree x2048 13.566 ms/op 13.241 ms/op 1.02
bytes32 toHexString 1.1450 us/op 1.0630 us/op 1.08
bytes32 Buffer.toString(hex) 796.00 ns/op 729.00 ns/op 1.09
bytes32 Buffer.toString(hex) from Uint8Array 1.0490 us/op 898.00 ns/op 1.17
bytes32 Buffer.toString(hex) + 0x 818.00 ns/op 724.00 ns/op 1.13
Object access 1 prop 0.39200 ns/op 0.37100 ns/op 1.06
Map access 1 prop 0.34100 ns/op 0.29000 ns/op 1.18
Object get x1000 20.938 ns/op 17.765 ns/op 1.18
Map get x1000 1.1390 ns/op 0.97000 ns/op 1.17
Object set x1000 126.82 ns/op 122.35 ns/op 1.04
Map set x1000 76.619 ns/op 72.236 ns/op 1.06
Return object 10000 times 0.42300 ns/op 0.37270 ns/op 1.13
Throw Error 10000 times 6.6892 us/op 5.8985 us/op 1.13
enrSubnets - fastDeserialize 64 bits 2.7940 us/op 2.6360 us/op 1.06
enrSubnets - ssz BitVector 64 bits 825.00 ns/op 755.00 ns/op 1.09
enrSubnets - fastDeserialize 4 bits 409.00 ns/op 383.00 ns/op 1.07
enrSubnets - ssz BitVector 4 bits 849.00 ns/op 768.00 ns/op 1.11
prioritizePeers score -10:0 att 32-0.1 sync 2-0 101.79 us/op 94.536 us/op 1.08
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 149.39 us/op 136.52 us/op 1.09
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 250.47 us/op 232.43 us/op 1.08
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 452.18 us/op 394.04 us/op 1.15
prioritizePeers score 0:0 att 64-1 sync 4-1 516.42 us/op 460.43 us/op 1.12
RateTracker 1000000 limit, 1 obj count per request 225.94 ns/op 204.39 ns/op 1.11
RateTracker 1000000 limit, 2 obj count per request 170.81 ns/op 156.54 ns/op 1.09
RateTracker 1000000 limit, 4 obj count per request 145.89 ns/op 133.34 ns/op 1.09
RateTracker 1000000 limit, 8 obj count per request 136.81 ns/op 120.23 ns/op 1.14
RateTracker with prune 4.4080 us/op 4.3290 us/op 1.02
array of 16000 items push then shift 3.5902 us/op 3.1887 us/op 1.13
LinkedList of 16000 items push then shift 19.229 ns/op 17.325 ns/op 1.11
array of 16000 items push then pop 252.30 ns/op 243.30 ns/op 1.04
LinkedList of 16000 items push then pop 18.408 ns/op 16.558 ns/op 1.11
array of 24000 items push then shift 5.0440 us/op 4.5567 us/op 1.11
LinkedList of 24000 items push then shift 21.066 ns/op 20.969 ns/op 1.00
array of 24000 items push then pop 231.67 ns/op 207.53 ns/op 1.12
LinkedList of 24000 items push then pop 19.730 ns/op 17.961 ns/op 1.10
intersect bitArray bitLen 8 12.852 ns/op 11.735 ns/op 1.10
intersect array and set length 8 174.58 ns/op 175.89 ns/op 0.99
intersect bitArray bitLen 128 78.901 ns/op 61.972 ns/op 1.27
intersect array and set length 128 2.4200 us/op 2.2845 us/op 1.06
Buffer.concat 32 items 2.0690 ns/op 1.9080 ns/op 1.08
pass gossip attestations to forkchoice per slot 3.5478 ms/op 3.1690 ms/op 1.12
computeDeltas 3.5213 ms/op 3.2744 ms/op 1.08
computeProposerBoostScoreFromBalances 1.0669 ms/op 921.21 us/op 1.16
altair processAttestation - 250000 vs - 7PWei normalcase 4.3898 ms/op 3.9620 ms/op 1.11
altair processAttestation - 250000 vs - 7PWei worstcase 6.5781 ms/op 5.8081 ms/op 1.13
altair processAttestation - setStatus - 1/6 committees join 224.48 us/op 207.47 us/op 1.08
altair processAttestation - setStatus - 1/3 committees join 437.86 us/op 399.61 us/op 1.10
altair processAttestation - setStatus - 1/2 committees join 606.06 us/op 565.89 us/op 1.07
altair processAttestation - setStatus - 2/3 committees join 852.84 us/op 720.15 us/op 1.18
altair processAttestation - setStatus - 4/5 committees join 1.1264 ms/op 1.0043 ms/op 1.12
altair processAttestation - setStatus - 100% committees join 1.3368 ms/op 1.2055 ms/op 1.11
altair processBlock - 250000 vs - 7PWei normalcase 29.885 ms/op 27.184 ms/op 1.10
altair processBlock - 250000 vs - 7PWei normalcase hashState 46.263 ms/op 41.131 ms/op 1.12
altair processBlock - 250000 vs - 7PWei worstcase 89.163 ms/op 80.569 ms/op 1.11
altair processBlock - 250000 vs - 7PWei worstcase hashState 110.35 ms/op 106.56 ms/op 1.04
phase0 processBlock - 250000 vs - 7PWei normalcase 3.9179 ms/op 3.5036 ms/op 1.12
phase0 processBlock - 250000 vs - 7PWei worstcase 52.996 ms/op 46.290 ms/op 1.14
altair processEth1Data - 250000 vs - 7PWei normalcase 946.44 us/op 820.57 us/op 1.15
Tree 40 250000 create 873.13 ms/op 810.92 ms/op 1.08
Tree 40 250000 get(125000) 322.62 ns/op 297.77 ns/op 1.08
Tree 40 250000 set(125000) 3.0327 us/op 2.8176 us/op 1.08
Tree 40 250000 toArray() 37.621 ms/op 32.972 ms/op 1.14
Tree 40 250000 iterate all - toArray() + loop 36.126 ms/op 33.027 ms/op 1.09
Tree 40 250000 iterate all - get(i) 123.17 ms/op 114.31 ms/op 1.08
MutableVector 250000 create 18.984 ms/op 18.275 ms/op 1.04
MutableVector 250000 get(125000) 14.682 ns/op 14.728 ns/op 1.00
MutableVector 250000 set(125000) 679.59 ns/op 637.26 ns/op 1.07
MutableVector 250000 toArray() 8.3797 ms/op 7.5972 ms/op 1.10
MutableVector 250000 iterate all - toArray() + loop 8.5996 ms/op 7.8426 ms/op 1.10
MutableVector 250000 iterate all - get(i) 3.7926 ms/op 3.3169 ms/op 1.14
Array 250000 create 7.0914 ms/op 7.1208 ms/op 1.00
Array 250000 clone - spread 4.2827 ms/op 3.8542 ms/op 1.11
Array 250000 get(125000) 1.7520 ns/op 1.5630 ns/op 1.12
Array 250000 set(125000) 1.7550 ns/op 1.5730 ns/op 1.12
Array 250000 iterate all - loop 184.55 us/op 167.82 us/op 1.10
effectiveBalanceIncrements clone Uint8Array 300000 88.021 us/op 93.211 us/op 0.94
effectiveBalanceIncrements clone MutableVector 300000 1.4060 us/op 1.2150 us/op 1.16
effectiveBalanceIncrements rw all Uint8Array 300000 285.37 us/op 252.55 us/op 1.13
effectiveBalanceIncrements rw all MutableVector 300000 233.59 ms/op 215.21 ms/op 1.09
phase0 afterProcessEpoch - 250000 vs - 7PWei 202.73 ms/op 189.09 ms/op 1.07
phase0 beforeProcessEpoch - 250000 vs - 7PWei 75.809 ms/op 78.778 ms/op 0.96
altair processEpoch - mainnet_e81889 647.63 ms/op 584.31 ms/op 1.11
mainnet_e81889 - altair beforeProcessEpoch 167.14 ms/op 169.98 ms/op 0.98
mainnet_e81889 - altair processJustificationAndFinalization 43.181 us/op 28.351 us/op 1.52
mainnet_e81889 - altair processInactivityUpdates 11.748 ms/op 10.435 ms/op 1.13
mainnet_e81889 - altair processRewardsAndPenalties 105.41 ms/op 92.550 ms/op 1.14
mainnet_e81889 - altair processRegistryUpdates 11.042 us/op 5.8880 us/op 1.88
mainnet_e81889 - altair processSlashings 2.5130 us/op 1.4800 us/op 1.70
mainnet_e81889 - altair processEth1DataReset 2.5600 us/op 1.6780 us/op 1.53
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.7180 ms/op 2.5349 ms/op 1.07
mainnet_e81889 - altair processSlashingsReset 15.746 us/op 8.6300 us/op 1.82
mainnet_e81889 - altair processRandaoMixesReset 16.113 us/op 10.002 us/op 1.61
mainnet_e81889 - altair processHistoricalRootsUpdate 2.1990 us/op 1.2170 us/op 1.81
mainnet_e81889 - altair processParticipationFlagUpdates 7.9010 us/op 6.1090 us/op 1.29
mainnet_e81889 - altair processSyncCommitteeUpdates 1.7620 us/op 915.00 ns/op 1.93
mainnet_e81889 - altair afterProcessEpoch 229.40 ms/op 198.64 ms/op 1.15
phase0 processEpoch - mainnet_e58758 659.52 ms/op 528.62 ms/op 1.25
mainnet_e58758 - phase0 beforeProcessEpoch 256.21 ms/op 232.51 ms/op 1.10
mainnet_e58758 - phase0 processJustificationAndFinalization 39.340 us/op 28.052 us/op 1.40
mainnet_e58758 - phase0 processRewardsAndPenalties 140.13 ms/op 124.16 ms/op 1.13
mainnet_e58758 - phase0 processRegistryUpdates 21.778 us/op 15.741 us/op 1.38
mainnet_e58758 - phase0 processSlashings 2.0030 us/op 1.2120 us/op 1.65
mainnet_e58758 - phase0 processEth1DataReset 2.4140 us/op 1.2330 us/op 1.96
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.6015 ms/op 2.1469 ms/op 1.21
mainnet_e58758 - phase0 processSlashingsReset 11.925 us/op 7.1740 us/op 1.66
mainnet_e58758 - phase0 processRandaoMixesReset 15.605 us/op 9.7740 us/op 1.60
mainnet_e58758 - phase0 processHistoricalRootsUpdate 2.2900 us/op 1.3000 us/op 1.76
mainnet_e58758 - phase0 processParticipationRecordUpdates 14.497 us/op 9.1440 us/op 1.59
mainnet_e58758 - phase0 afterProcessEpoch 177.35 ms/op 162.19 ms/op 1.09
phase0 processEffectiveBalanceUpdates - 250000 normalcase 3.2235 ms/op 2.6799 ms/op 1.20
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 3.9574 ms/op 3.5106 ms/op 1.13
altair processInactivityUpdates - 250000 normalcase 44.892 ms/op 41.851 ms/op 1.07
altair processInactivityUpdates - 250000 worstcase 56.705 ms/op 50.464 ms/op 1.12
phase0 processRegistryUpdates - 250000 normalcase 21.753 us/op 11.197 us/op 1.94
phase0 processRegistryUpdates - 250000 badcase_full_deposits 640.14 us/op 455.30 us/op 1.41
phase0 processRegistryUpdates - 250000 worstcase 0.5 241.54 ms/op 218.12 ms/op 1.11
altair processRewardsAndPenalties - 250000 normalcase 143.52 ms/op 126.71 ms/op 1.13
altair processRewardsAndPenalties - 250000 worstcase 95.679 ms/op 86.801 ms/op 1.10
phase0 getAttestationDeltas - 250000 normalcase 15.240 ms/op 14.096 ms/op 1.08
phase0 getAttestationDeltas - 250000 worstcase 15.549 ms/op 13.876 ms/op 1.12
phase0 processSlashings - 250000 worstcase 6.1288 ms/op 5.3162 ms/op 1.15
altair processSyncCommitteeUpdates - 250000 313.09 ms/op 288.53 ms/op 1.09
BeaconState.hashTreeRoot - No change 571.00 ns/op 476.00 ns/op 1.20
BeaconState.hashTreeRoot - 1 full validator 71.995 us/op 64.167 us/op 1.12
BeaconState.hashTreeRoot - 32 full validator 823.19 us/op 658.14 us/op 1.25
BeaconState.hashTreeRoot - 512 full validator 6.4181 ms/op 7.7867 ms/op 0.82
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 86.169 us/op 79.502 us/op 1.08
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.2700 ms/op 1.1900 ms/op 1.07
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 17.427 ms/op 15.076 ms/op 1.16
BeaconState.hashTreeRoot - 1 balances 69.411 us/op 62.232 us/op 1.12
BeaconState.hashTreeRoot - 32 balances 670.18 us/op 584.73 us/op 1.15
BeaconState.hashTreeRoot - 512 balances 6.3902 ms/op 5.8584 ms/op 1.09
BeaconState.hashTreeRoot - 250000 balances 105.18 ms/op 94.557 ms/op 1.11
aggregationBits - 2048 els - zipIndexesInBitList 40.194 us/op 37.574 us/op 1.07
regular array get 100000 times 76.842 us/op 67.430 us/op 1.14
wrappedArray get 100000 times 74.448 us/op 67.535 us/op 1.10
arrayWithProxy get 100000 times 33.332 ms/op 32.466 ms/op 1.03
ssz.Root.equals 603.00 ns/op 484.00 ns/op 1.25
byteArrayEquals 479.00 ns/op 475.00 ns/op 1.01
shuffle list - 16384 els 12.119 ms/op 11.303 ms/op 1.07
shuffle list - 250000 els 183.48 ms/op 166.49 ms/op 1.10
processSlot - 1 slots 12.991 us/op 12.347 us/op 1.05
processSlot - 32 slots 1.9411 ms/op 1.7545 ms/op 1.11
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 425.93 us/op 383.03 us/op 1.11
getCommitteeAssignments - req 1 vs - 250000 vc 6.1588 ms/op 5.3516 ms/op 1.15
getCommitteeAssignments - req 100 vs - 250000 vc 8.7037 ms/op 7.3395 ms/op 1.19
getCommitteeAssignments - req 1000 vs - 250000 vc 8.8844 ms/op 7.8250 ms/op 1.14
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 12.640 ns/op 9.5400 ns/op 1.32
state getBlockRootAtSlot - 250000 vs - 7PWei 1.2650 us/op 1.1926 us/op 1.06
computeProposers - vc 250000 18.400 ms/op 17.288 ms/op 1.06
computeEpochShuffling - vc 250000 183.72 ms/op 170.73 ms/op 1.08
getNextSyncCommittee - vc 250000 298.66 ms/op 283.53 ms/op 1.05

by benchmarkbot/action

wemeetagain
wemeetagain previously approved these changes Aug 24, 2022
View reviewed changes
@wemeetagain
Copy link
Member

looks great

some fork choice tests fail https://github.com/ChainSafe/lodestar/runs/7995278656?check_suite_focus=true#step:16:1603

@wemeetagain wemeetagain merged commit 8654b9d into unstable Sep 1, 2022
@wemeetagain wemeetagain deleted the dapplion/review-metrics branch September 1, 2022 16:48
@dapplion dapplion mentioned this pull request Sep 7, 2022
Merged
@nflaig nflaig mentioned this pull request May 17, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes

Assignees
No one assigned
Labels
scope-metrics All issues with regards to the exposed metrics.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dapplion @wemeetagain