Add block publish delay to validator monitor dashboard

[wemeetagain]

Motivation

We need to know to what extent Lodestar is publishing blocks late, so we can fix it.

Description

Add this panel to the validator monitor dashboard:

[g11tech]

lgtm

[github-actions]

Performance Report

✔️ no performance regression detected

Full benchmark results

Benchmark suite	Current: b328e41	Previous: 16cbdc3	Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	663.57 us/op	758.95 us/op	0.87
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	74.923 us/op	63.535 us/op	1.18
BLS verify - blst-native	1.2990 ms/op	1.3587 ms/op	0.96
BLS verifyMultipleSignatures 3 - blst-native	2.6385 ms/op	2.7650 ms/op	0.95
BLS verifyMultipleSignatures 8 - blst-native	5.7175 ms/op	5.7587 ms/op	0.99
BLS verifyMultipleSignatures 32 - blst-native	20.610 ms/op	21.259 ms/op	0.97
BLS aggregatePubkeys 32 - blst-native	28.637 us/op	27.823 us/op	1.03
BLS aggregatePubkeys 128 - blst-native	108.58 us/op	108.51 us/op	1.00
getAttestationsForBlock	89.405 ms/op	66.796 ms/op	1.34
isKnown best case - 1 super set check	316.00 ns/op	314.00 ns/op	1.01
isKnown normal case - 2 super set checks	294.00 ns/op	293.00 ns/op	1.00
isKnown worse case - 16 super set checks	301.00 ns/op	303.00 ns/op	0.99
CheckpointStateCache - add get delete	7.0420 us/op	6.7710 us/op	1.04
validate gossip signedAggregateAndProof - struct	3.0378 ms/op	3.0161 ms/op	1.01
validate gossip attestation - struct	1.4079 ms/op	1.5227 ms/op	0.92
pickEth1Vote - no votes	1.6278 ms/op	1.4517 ms/op	1.12
pickEth1Vote - max votes	15.789 ms/op	11.853 ms/op	1.33
pickEth1Vote - Eth1Data hashTreeRoot value x2048	10.703 ms/op	10.016 ms/op	1.07
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	21.402 ms/op	15.774 ms/op	1.36
pickEth1Vote - Eth1Data fastSerialize value x2048	869.12 us/op	777.20 us/op	1.12
pickEth1Vote - Eth1Data fastSerialize tree x2048	6.2401 ms/op	5.7704 ms/op	1.08
bytes32 toHexString	783.00 ns/op	739.00 ns/op	1.06
bytes32 Buffer.toString(hex)	458.00 ns/op	450.00 ns/op	1.02
bytes32 Buffer.toString(hex) from Uint8Array	667.00 ns/op	658.00 ns/op	1.01
bytes32 Buffer.toString(hex) + 0x	463.00 ns/op	455.00 ns/op	1.02
Object access 1 prop	0.21800 ns/op	0.21300 ns/op	1.02
Map access 1 prop	0.19100 ns/op	0.18500 ns/op	1.03
Object get x1000	7.7260 ns/op	7.6630 ns/op	1.01
Map get x1000	0.59200 ns/op	0.71700 ns/op	0.83
Object set x1000	60.088 ns/op	78.190 ns/op	0.77
Map set x1000	48.736 ns/op	60.474 ns/op	0.81
Return object 10000 times	0.29620 ns/op	0.28370 ns/op	1.04
Throw Error 10000 times	5.1507 us/op	4.7841 us/op	1.08
fastMsgIdFn sha256 / 200 bytes	3.7740 us/op	3.9730 us/op	0.95
fastMsgIdFn h32 xxhash / 200 bytes	334.00 ns/op	346.00 ns/op	0.97
fastMsgIdFn h64 xxhash / 200 bytes	540.00 ns/op	499.00 ns/op	1.08
fastMsgIdFn sha256 / 1000 bytes	12.849 us/op	12.617 us/op	1.02
fastMsgIdFn h32 xxhash / 1000 bytes	520.00 ns/op	510.00 ns/op	1.02
fastMsgIdFn h64 xxhash / 1000 bytes	719.00 ns/op	578.00 ns/op	1.24
fastMsgIdFn sha256 / 10000 bytes	119.93 us/op	114.09 us/op	1.05
fastMsgIdFn h32 xxhash / 10000 bytes	2.3030 us/op	2.1530 us/op	1.07
fastMsgIdFn h64 xxhash / 10000 bytes	1.5830 us/op	1.5550 us/op	1.02
enrSubnets - fastDeserialize 64 bits	2.4700 us/op	1.7020 us/op	1.45
enrSubnets - ssz BitVector 64 bits	805.00 ns/op	569.00 ns/op	1.41
enrSubnets - fastDeserialize 4 bits	249.00 ns/op	209.00 ns/op	1.19
enrSubnets - ssz BitVector 4 bits	708.00 ns/op	607.00 ns/op	1.17
prioritizePeers score -10:0 att 32-0.1 sync 2-0	148.33 us/op	105.39 us/op	1.41
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	205.60 us/op	151.36 us/op	1.36
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	251.76 us/op	211.56 us/op	1.19
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	410.19 us/op	381.49 us/op	1.08
prioritizePeers score 0:0 att 64-1 sync 4-1	462.38 us/op	415.67 us/op	1.11
array of 16000 items push then shift	1.7829 us/op	1.8036 us/op	0.99
LinkedList of 16000 items push then shift	10.664 ns/op	9.5290 ns/op	1.12
array of 16000 items push then pop	133.33 ns/op	121.48 ns/op	1.10
LinkedList of 16000 items push then pop	10.296 ns/op	9.4780 ns/op	1.09
array of 24000 items push then shift	2.5998 us/op	2.4796 us/op	1.05
LinkedList of 24000 items push then shift	11.025 ns/op	10.372 ns/op	1.06
array of 24000 items push then pop	105.69 ns/op	97.988 ns/op	1.08
LinkedList of 24000 items push then pop	10.305 ns/op	9.5220 ns/op	1.08
intersect bitArray bitLen 8	14.809 ns/op	14.912 ns/op	0.99
intersect array and set length 8	109.70 ns/op	92.938 ns/op	1.18
intersect bitArray bitLen 128	48.936 ns/op	48.704 ns/op	1.00
intersect array and set length 128	1.3905 us/op	1.3825 us/op	1.01
Buffer.concat 32 items	3.4840 us/op	2.9580 us/op	1.18
Uint8Array.set 32 items	2.5840 us/op	2.5620 us/op	1.01
pass gossip attestations to forkchoice per slot	4.2217 ms/op	4.5761 ms/op	0.92
computeDeltas	4.0374 ms/op	3.3136 ms/op	1.22
computeProposerBoostScoreFromBalances	1.9945 ms/op	1.9749 ms/op	1.01
altair processAttestation - 250000 vs - 7PWei normalcase	3.5534 ms/op	3.6696 ms/op	0.97
altair processAttestation - 250000 vs - 7PWei worstcase	4.4836 ms/op	3.7179 ms/op	1.21
altair processAttestation - setStatus - 1/6 committees join	171.04 us/op	158.00 us/op	1.08
altair processAttestation - setStatus - 1/3 committees join	302.97 us/op	298.06 us/op	1.02
altair processAttestation - setStatus - 1/2 committees join	403.14 us/op	410.02 us/op	0.98
altair processAttestation - setStatus - 2/3 committees join	516.26 us/op	530.24 us/op	0.97
altair processAttestation - setStatus - 4/5 committees join	785.28 us/op	723.83 us/op	1.08
altair processAttestation - setStatus - 100% committees join	988.80 us/op	888.60 us/op	1.11
altair processBlock - 250000 vs - 7PWei normalcase	23.521 ms/op	20.816 ms/op	1.13
altair processBlock - 250000 vs - 7PWei normalcase hashState	38.436 ms/op	29.613 ms/op	1.30
altair processBlock - 250000 vs - 7PWei worstcase	65.481 ms/op	55.740 ms/op	1.17
altair processBlock - 250000 vs - 7PWei worstcase hashState	85.511 ms/op	78.266 ms/op	1.09
phase0 processBlock - 250000 vs - 7PWei normalcase	3.7148 ms/op	2.2544 ms/op	1.65
phase0 processBlock - 250000 vs - 7PWei worstcase	39.142 ms/op	32.791 ms/op	1.19
altair processEth1Data - 250000 vs - 7PWei normalcase	684.29 us/op	579.22 us/op	1.18
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15	14.589 us/op	10.331 us/op	1.41
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219	38.453 us/op	30.906 us/op	1.24
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42	18.369 us/op	13.425 us/op	1.37
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18	14.492 us/op	11.138 us/op	1.30
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020	132.26 us/op	96.592 us/op	1.37
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777	1.2436 ms/op	814.35 us/op	1.53
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384	1.7685 ms/op	931.05 us/op	1.90
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384	1.5283 ms/op	960.27 us/op	1.59
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384	6.2930 ms/op	3.3925 ms/op	1.85
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384	3.2186 ms/op	1.9337 ms/op	1.66
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384	8.6904 ms/op	5.3812 ms/op	1.61
Tree 40 250000 create	879.61 ms/op	490.33 ms/op	1.79
Tree 40 250000 get(125000)	257.07 ns/op	210.64 ns/op	1.22
Tree 40 250000 set(125000)	3.2419 us/op	1.1673 us/op	2.78
Tree 40 250000 toArray()	35.837 ms/op	22.845 ms/op	1.57
Tree 40 250000 iterate all - toArray() + loop	33.496 ms/op	26.692 ms/op	1.25
Tree 40 250000 iterate all - get(i)	101.42 ms/op	82.970 ms/op	1.22
MutableVector 250000 create	14.799 ms/op	12.027 ms/op	1.23
MutableVector 250000 get(125000)	8.1900 ns/op	7.1140 ns/op	1.15
MutableVector 250000 set(125000)	719.68 ns/op	301.78 ns/op	2.38
MutableVector 250000 toArray()	6.0679 ms/op	3.8666 ms/op	1.57
MutableVector 250000 iterate all - toArray() + loop	5.1094 ms/op	4.2231 ms/op	1.21
MutableVector 250000 iterate all - get(i)	2.3371 ms/op	1.6605 ms/op	1.41
Array 250000 create	5.3899 ms/op	3.3802 ms/op	1.59
Array 250000 clone - spread	1.8039 ms/op	1.5578 ms/op	1.16
Array 250000 get(125000)	1.4880 ns/op	0.79000 ns/op	1.88
Array 250000 set(125000)	1.6800 ns/op	0.82600 ns/op	2.03
Array 250000 iterate all - loop	128.65 us/op	109.33 us/op	1.18
effectiveBalanceIncrements clone Uint8Array 300000	56.671 us/op	62.414 us/op	0.91
effectiveBalanceIncrements clone MutableVector 300000	400.00 ns/op	629.00 ns/op	0.64
effectiveBalanceIncrements rw all Uint8Array 300000	211.63 us/op	194.19 us/op	1.09
effectiveBalanceIncrements rw all MutableVector 300000	200.43 ms/op	176.58 ms/op	1.14
phase0 afterProcessEpoch - 250000 vs - 7PWei	141.28 ms/op	135.27 ms/op	1.04
phase0 beforeProcessEpoch - 250000 vs - 7PWei	89.868 ms/op	50.183 ms/op	1.79
altair processEpoch - mainnet_e81889	439.71 ms/op	386.82 ms/op	1.14
mainnet_e81889 - altair beforeProcessEpoch	82.948 ms/op	93.102 ms/op	0.89
mainnet_e81889 - altair processJustificationAndFinalization	27.081 us/op	24.485 us/op	1.11
mainnet_e81889 - altair processInactivityUpdates	8.9509 ms/op	7.1610 ms/op	1.25
mainnet_e81889 - altair processRewardsAndPenalties	85.225 ms/op	85.036 ms/op	1.00
mainnet_e81889 - altair processRegistryUpdates	5.6030 us/op	6.4540 us/op	0.87
mainnet_e81889 - altair processSlashings	957.00 ns/op	1.5870 us/op	0.60
mainnet_e81889 - altair processEth1DataReset	1.0850 us/op	1.7520 us/op	0.62
mainnet_e81889 - altair processEffectiveBalanceUpdates	1.9107 ms/op	1.5951 ms/op	1.20
mainnet_e81889 - altair processSlashingsReset	10.513 us/op	7.0560 us/op	1.49
mainnet_e81889 - altair processRandaoMixesReset	6.5550 us/op	9.2900 us/op	0.71
mainnet_e81889 - altair processHistoricalRootsUpdate	1.3540 us/op	1.1880 us/op	1.14
mainnet_e81889 - altair processParticipationFlagUpdates	9.0970 us/op	7.8830 us/op	1.15
mainnet_e81889 - altair processSyncCommitteeUpdates	1.0550 us/op	1.8650 us/op	0.57
mainnet_e81889 - altair afterProcessEpoch	139.48 ms/op	149.85 ms/op	0.93
phase0 processEpoch - mainnet_e58758	455.90 ms/op	501.71 ms/op	0.91
mainnet_e58758 - phase0 beforeProcessEpoch	178.70 ms/op	199.92 ms/op	0.89
mainnet_e58758 - phase0 processJustificationAndFinalization	35.331 us/op	38.407 us/op	0.92
mainnet_e58758 - phase0 processRewardsAndPenalties	86.763 ms/op	81.603 ms/op	1.06
mainnet_e58758 - phase0 processRegistryUpdates	12.861 us/op	19.631 us/op	0.66
mainnet_e58758 - phase0 processSlashings	1.1050 us/op	1.4260 us/op	0.77
mainnet_e58758 - phase0 processEth1DataReset	1.2230 us/op	1.1230 us/op	1.09
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	1.1493 ms/op	1.2085 ms/op	0.95
mainnet_e58758 - phase0 processSlashingsReset	6.9860 us/op	6.1260 us/op	1.14
mainnet_e58758 - phase0 processRandaoMixesReset	5.8100 us/op	5.5360 us/op	1.05
mainnet_e58758 - phase0 processHistoricalRootsUpdate	677.00 ns/op	1.3640 us/op	0.50
mainnet_e58758 - phase0 processParticipationRecordUpdates	6.6030 us/op	15.424 us/op	0.43
mainnet_e58758 - phase0 afterProcessEpoch	103.83 ms/op	114.70 ms/op	0.91
phase0 processEffectiveBalanceUpdates - 250000 normalcase	1.3017 ms/op	1.4568 ms/op	0.89
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	1.7219 ms/op	1.7548 ms/op	0.98
altair processInactivityUpdates - 250000 normalcase	23.986 ms/op	22.657 ms/op	1.06
altair processInactivityUpdates - 250000 worstcase	29.735 ms/op	28.392 ms/op	1.05
phase0 processRegistryUpdates - 250000 normalcase	11.992 us/op	8.0320 us/op	1.49
phase0 processRegistryUpdates - 250000 badcase_full_deposits	341.78 us/op	303.92 us/op	1.12
phase0 processRegistryUpdates - 250000 worstcase 0.5	144.37 ms/op	140.59 ms/op	1.03
altair processRewardsAndPenalties - 250000 normalcase	74.842 ms/op	71.657 ms/op	1.04
altair processRewardsAndPenalties - 250000 worstcase	71.883 ms/op	73.625 ms/op	0.98
phase0 getAttestationDeltas - 250000 normalcase	7.3912 ms/op	8.8770 ms/op	0.83
phase0 getAttestationDeltas - 250000 worstcase	8.7896 ms/op	8.3605 ms/op	1.05
phase0 processSlashings - 250000 worstcase	3.9166 ms/op	4.2003 ms/op	0.93
altair processSyncCommitteeUpdates - 250000	198.29 ms/op	194.56 ms/op	1.02
BeaconState.hashTreeRoot - No change	297.00 ns/op	304.00 ns/op	0.98
BeaconState.hashTreeRoot - 1 full validator	54.530 us/op	56.311 us/op	0.97
BeaconState.hashTreeRoot - 32 full validator	588.54 us/op	628.36 us/op	0.94
BeaconState.hashTreeRoot - 512 full validator	7.0347 ms/op	6.2751 ms/op	1.12
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	67.429 us/op	72.540 us/op	0.93
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	965.55 us/op	1.0292 ms/op	0.94
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	13.252 ms/op	12.731 ms/op	1.04
BeaconState.hashTreeRoot - 1 balances	56.081 us/op	59.282 us/op	0.95
BeaconState.hashTreeRoot - 32 balances	461.17 us/op	528.81 us/op	0.87
BeaconState.hashTreeRoot - 512 balances	5.5343 ms/op	5.2234 ms/op	1.06
BeaconState.hashTreeRoot - 250000 balances	79.736 ms/op	89.174 ms/op	0.89
aggregationBits - 2048 els - zipIndexesInBitList	17.927 us/op	25.520 us/op	0.70
regular array get 100000 times	34.522 us/op	50.290 us/op	0.69
wrappedArray get 100000 times	34.367 us/op	38.162 us/op	0.90
arrayWithProxy get 100000 times	17.116 ms/op	21.308 ms/op	0.80
ssz.Root.equals	604.00 ns/op	664.00 ns/op	0.91
byteArrayEquals	605.00 ns/op	627.00 ns/op	0.96
shuffle list - 16384 els	7.3103 ms/op	7.6816 ms/op	0.95
shuffle list - 250000 els	108.16 ms/op	111.56 ms/op	0.97
processSlot - 1 slots	10.135 us/op	10.520 us/op	0.96
processSlot - 32 slots	1.4837 ms/op	1.5431 ms/op	0.96
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	216.47 us/op	242.59 us/op	0.89
getCommitteeAssignments - req 1 vs - 250000 vc	3.0578 ms/op	3.2850 ms/op	0.93
getCommitteeAssignments - req 100 vs - 250000 vc	4.3299 ms/op	4.7983 ms/op	0.90
getCommitteeAssignments - req 1000 vs - 250000 vc	4.6564 ms/op	5.2380 ms/op	0.89
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei	6.1000 ns/op	5.8700 ns/op	1.04
state getBlockRootAtSlot - 250000 vs - 7PWei	1.1146 us/op	830.11 ns/op	1.34
computeProposers - vc 250000	13.402 ms/op	13.555 ms/op	0.99
computeEpochShuffling - vc 250000	121.23 ms/op	117.92 ms/op	1.03
getNextSyncCommittee - vc 250000	198.70 ms/op	219.83 ms/op	0.90

by benchmarkbot/action

[wemeetagain]

🎉 This PR is included in v1.6.0 🎉