Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Track gossipsub iwant promises that never delivered percentage #5092

Merged
merged 1 commit into from
Feb 7, 2023
Merged

Track gossipsub iwant promises that never delivered percentage #5092

merged 1 commit into from
Feb 7, 2023

Conversation

twoeths
Copy link
Contributor

@twoeths twoeths commented Feb 2, 2023

Motivation

Iwant promises that's never delivered are not tracked correctly because we only keep iwantPromiseDeliveryTime items in gossipsub tracer up to 120s then they are pruned

Description

  • Correct the formation: 1 - sum(rate(gossipsub_iwant_promise_delivery_seconds_count[$__rate_interval])) / sum(rate(gossipsub_iwant_promise_sent_total[$__rate_interval]))
  • It shows from Jan 25 there are a lot of Iwant promises are never delivered on mainnet nodes:

Screen Shot 2023-02-02 at 13 51 25

part of #5074

@github-actions
Copy link
Contributor

github-actions bot commented Feb 2, 2023 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: ad692e9 Previous: 6793305 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 932.28 us/op 518.29 us/op 1.80
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 44.727 us/op 46.398 us/op 0.96
BLS verify - blst-native 1.2115 ms/op 1.2119 ms/op 1.00
BLS verifyMultipleSignatures 3 - blst-native 2.4648 ms/op 2.4720 ms/op 1.00
BLS verifyMultipleSignatures 8 - blst-native 5.2898 ms/op 5.3799 ms/op 0.98
BLS verifyMultipleSignatures 32 - blst-native 19.099 ms/op 19.388 ms/op 0.99
BLS aggregatePubkeys 32 - blst-native 25.540 us/op 26.034 us/op 0.98
BLS aggregatePubkeys 128 - blst-native 99.730 us/op 101.77 us/op 0.98
getAttestationsForBlock 53.228 ms/op 55.973 ms/op 0.95
isKnown best case - 1 super set check 272.00 ns/op 274.00 ns/op 0.99
isKnown normal case - 2 super set checks 266.00 ns/op 264.00 ns/op 1.01
isKnown worse case - 16 super set checks 265.00 ns/op 263.00 ns/op 1.01
CheckpointStateCache - add get delete 5.1700 us/op 6.0110 us/op 0.86
validate gossip signedAggregateAndProof - struct 2.7612 ms/op 2.7958 ms/op 0.99
validate gossip attestation - struct 1.3179 ms/op 1.3385 ms/op 0.98
pickEth1Vote - no votes 1.2385 ms/op 1.3523 ms/op 0.92
pickEth1Vote - max votes 10.504 ms/op 9.7881 ms/op 1.07
pickEth1Vote - Eth1Data hashTreeRoot value x2048 8.7370 ms/op 8.8528 ms/op 0.99
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 14.601 ms/op 14.371 ms/op 1.02
pickEth1Vote - Eth1Data fastSerialize value x2048 629.98 us/op 662.21 us/op 0.95
pickEth1Vote - Eth1Data fastSerialize tree x2048 7.9460 ms/op 4.7295 ms/op 1.68
bytes32 toHexString 486.00 ns/op 503.00 ns/op 0.97
bytes32 Buffer.toString(hex) 336.00 ns/op 354.00 ns/op 0.95
bytes32 Buffer.toString(hex) from Uint8Array 525.00 ns/op 576.00 ns/op 0.91
bytes32 Buffer.toString(hex) + 0x 334.00 ns/op 348.00 ns/op 0.96
Object access 1 prop 0.16600 ns/op 0.17600 ns/op 0.94
Map access 1 prop 0.15200 ns/op 0.16500 ns/op 0.92
Object get x1000 6.4610 ns/op 6.9600 ns/op 0.93
Map get x1000 0.60600 ns/op 0.64800 ns/op 0.94
Object set x1000 50.363 ns/op 55.808 ns/op 0.90
Map set x1000 41.434 ns/op 45.509 ns/op 0.91
Return object 10000 times 0.22750 ns/op 0.24060 ns/op 0.95
Throw Error 10000 times 3.9608 us/op 4.1654 us/op 0.95
fastMsgIdFn sha256 / 200 bytes 3.2710 us/op 3.5030 us/op 0.93
fastMsgIdFn h32 xxhash / 200 bytes 261.00 ns/op 294.00 ns/op 0.89
fastMsgIdFn h64 xxhash / 200 bytes 382.00 ns/op 407.00 ns/op 0.94
fastMsgIdFn sha256 / 1000 bytes 11.758 us/op 11.786 us/op 1.00
fastMsgIdFn h32 xxhash / 1000 bytes 431.00 ns/op 435.00 ns/op 0.99
fastMsgIdFn h64 xxhash / 1000 bytes 486.00 ns/op 490.00 ns/op 0.99
fastMsgIdFn sha256 / 10000 bytes 104.67 us/op 104.91 us/op 1.00
fastMsgIdFn h32 xxhash / 10000 bytes 1.9780 us/op 1.9850 us/op 1.00
fastMsgIdFn h64 xxhash / 10000 bytes 1.3930 us/op 1.3830 us/op 1.01
enrSubnets - fastDeserialize 64 bits 1.2800 us/op 1.2830 us/op 1.00
enrSubnets - ssz BitVector 64 bits 475.00 ns/op 490.00 ns/op 0.97
enrSubnets - fastDeserialize 4 bits 165.00 ns/op 176.00 ns/op 0.94
enrSubnets - ssz BitVector 4 bits 479.00 ns/op 496.00 ns/op 0.97
prioritizePeers score -10:0 att 32-0.1 sync 2-0 94.796 us/op 97.228 us/op 0.97
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 126.93 us/op 134.17 us/op 0.95
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 170.13 us/op 190.41 us/op 0.89
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 305.19 us/op 362.76 us/op 0.84
prioritizePeers score 0:0 att 64-1 sync 4-1 356.02 us/op 414.90 us/op 0.86
array of 16000 items push then shift 1.6312 us/op 1.6785 us/op 0.97
LinkedList of 16000 items push then shift 8.7650 ns/op 9.1830 ns/op 0.95
array of 16000 items push then pop 73.122 ns/op 115.78 ns/op 0.63
LinkedList of 16000 items push then pop 8.5120 ns/op 9.0360 ns/op 0.94
array of 24000 items push then shift 2.3382 us/op 2.3943 us/op 0.98
LinkedList of 24000 items push then shift 8.7150 ns/op 9.2490 ns/op 0.94
array of 24000 items push then pop 74.915 ns/op 85.502 ns/op 0.88
LinkedList of 24000 items push then pop 8.6010 ns/op 9.0480 ns/op 0.95
intersect bitArray bitLen 8 12.956 ns/op 13.187 ns/op 0.98
intersect array and set length 8 80.703 ns/op 89.776 ns/op 0.90
intersect bitArray bitLen 128 45.071 ns/op 44.007 ns/op 1.02
intersect array and set length 128 1.0706 us/op 1.2529 us/op 0.85
Buffer.concat 32 items 2.7310 us/op 2.7550 us/op 0.99
Uint8Array.set 32 items 2.7830 us/op 2.3090 us/op 1.21
pass gossip attestations to forkchoice per slot 2.3625 ms/op 4.1862 ms/op 0.56
computeDeltas 2.9960 ms/op 3.0512 ms/op 0.98
computeProposerBoostScoreFromBalances 1.8112 ms/op 1.8142 ms/op 1.00
altair processAttestation - 250000 vs - 7PWei normalcase 2.8987 ms/op 2.5599 ms/op 1.13
altair processAttestation - 250000 vs - 7PWei worstcase 3.8755 ms/op 3.9573 ms/op 0.98
altair processAttestation - setStatus - 1/6 committees join 147.42 us/op 140.25 us/op 1.05
altair processAttestation - setStatus - 1/3 committees join 284.65 us/op 279.63 us/op 1.02
altair processAttestation - setStatus - 1/2 committees join 382.34 us/op 372.66 us/op 1.03
altair processAttestation - setStatus - 2/3 committees join 490.64 us/op 464.64 us/op 1.06
altair processAttestation - setStatus - 4/5 committees join 678.97 us/op 646.82 us/op 1.05
altair processAttestation - setStatus - 100% committees join 773.20 us/op 759.52 us/op 1.02
altair processBlock - 250000 vs - 7PWei normalcase 17.731 ms/op 16.388 ms/op 1.08
altair processBlock - 250000 vs - 7PWei normalcase hashState 26.103 ms/op 24.709 ms/op 1.06
altair processBlock - 250000 vs - 7PWei worstcase 48.707 ms/op 49.659 ms/op 0.98
altair processBlock - 250000 vs - 7PWei worstcase hashState 69.903 ms/op 63.062 ms/op 1.11
phase0 processBlock - 250000 vs - 7PWei normalcase 2.4450 ms/op 1.9034 ms/op 1.28
phase0 processBlock - 250000 vs - 7PWei worstcase 30.639 ms/op 27.749 ms/op 1.10
altair processEth1Data - 250000 vs - 7PWei normalcase 526.01 us/op 446.19 us/op 1.18
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 8.1000 us/op 6.9670 us/op 1.16
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 25.777 us/op 20.343 us/op 1.27
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 11.035 us/op 8.2660 us/op 1.33
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 7.2880 us/op 7.1890 us/op 1.01
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 89.053 us/op 80.662 us/op 1.10
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 642.85 us/op 636.55 us/op 1.01
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 884.07 us/op 894.89 us/op 0.99
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 853.40 us/op 854.96 us/op 1.00
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.3434 ms/op 2.2896 ms/op 1.02
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.4667 ms/op 1.5219 ms/op 0.96
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 4.2665 ms/op 3.8296 ms/op 1.11
Tree 40 250000 create 338.89 ms/op 289.24 ms/op 1.17
Tree 40 250000 get(125000) 198.90 ns/op 177.83 ns/op 1.12
Tree 40 250000 set(125000) 1.0386 us/op 889.25 ns/op 1.17
Tree 40 250000 toArray() 21.893 ms/op 17.563 ms/op 1.25
Tree 40 250000 iterate all - toArray() + loop 21.602 ms/op 17.659 ms/op 1.22
Tree 40 250000 iterate all - get(i) 74.417 ms/op 65.891 ms/op 1.13
MutableVector 250000 create 10.564 ms/op 10.877 ms/op 0.97
MutableVector 250000 get(125000) 6.2540 ns/op 6.2750 ns/op 1.00
MutableVector 250000 set(125000) 272.50 ns/op 279.07 ns/op 0.98
MutableVector 250000 toArray() 3.1961 ms/op 3.0755 ms/op 1.04
MutableVector 250000 iterate all - toArray() + loop 3.4976 ms/op 3.4278 ms/op 1.02
MutableVector 250000 iterate all - get(i) 1.5228 ms/op 1.4983 ms/op 1.02
Array 250000 create 3.0943 ms/op 2.7139 ms/op 1.14
Array 250000 clone - spread 1.2579 ms/op 1.0748 ms/op 1.17
Array 250000 get(125000) 0.61700 ns/op 0.52800 ns/op 1.17
Array 250000 set(125000) 0.69000 ns/op 0.59800 ns/op 1.15
Array 250000 iterate all - loop 111.25 us/op 109.33 us/op 1.02
effectiveBalanceIncrements clone Uint8Array 300000 37.672 us/op 27.912 us/op 1.35
effectiveBalanceIncrements clone MutableVector 300000 396.00 ns/op 314.00 ns/op 1.26
effectiveBalanceIncrements rw all Uint8Array 300000 168.32 us/op 164.46 us/op 1.02
effectiveBalanceIncrements rw all MutableVector 300000 91.618 ms/op 76.221 ms/op 1.20
phase0 afterProcessEpoch - 250000 vs - 7PWei 117.09 ms/op 110.98 ms/op 1.06
phase0 beforeProcessEpoch - 250000 vs - 7PWei 36.807 ms/op 43.363 ms/op 0.85
altair processEpoch - mainnet_e81889 321.56 ms/op 310.90 ms/op 1.03
mainnet_e81889 - altair beforeProcessEpoch 66.108 ms/op 52.134 ms/op 1.27
mainnet_e81889 - altair processJustificationAndFinalization 17.714 us/op 19.173 us/op 0.92
mainnet_e81889 - altair processInactivityUpdates 6.6386 ms/op 5.8954 ms/op 1.13
mainnet_e81889 - altair processRewardsAndPenalties 72.528 ms/op 68.933 ms/op 1.05
mainnet_e81889 - altair processRegistryUpdates 2.9060 us/op 2.5200 us/op 1.15
mainnet_e81889 - altair processSlashings 486.00 ns/op 871.00 ns/op 0.56
mainnet_e81889 - altair processEth1DataReset 604.00 ns/op 566.00 ns/op 1.07
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.2647 ms/op 1.2343 ms/op 1.02
mainnet_e81889 - altair processSlashingsReset 4.1170 us/op 4.9820 us/op 0.83
mainnet_e81889 - altair processRandaoMixesReset 7.9170 us/op 6.4100 us/op 1.24
mainnet_e81889 - altair processHistoricalRootsUpdate 932.00 ns/op 1.0560 us/op 0.88
mainnet_e81889 - altair processParticipationFlagUpdates 2.3050 us/op 3.0650 us/op 0.75
mainnet_e81889 - altair processSyncCommitteeUpdates 534.00 ns/op 831.00 ns/op 0.64
mainnet_e81889 - altair afterProcessEpoch 131.88 ms/op 129.64 ms/op 1.02
phase0 processEpoch - mainnet_e58758 340.73 ms/op 453.30 ms/op 0.75
mainnet_e58758 - phase0 beforeProcessEpoch 124.80 ms/op 139.36 ms/op 0.90
mainnet_e58758 - phase0 processJustificationAndFinalization 19.488 us/op 17.670 us/op 1.10
mainnet_e58758 - phase0 processRewardsAndPenalties 59.461 ms/op 65.780 ms/op 0.90
mainnet_e58758 - phase0 processRegistryUpdates 7.4440 us/op 9.9490 us/op 0.75
mainnet_e58758 - phase0 processSlashings 511.00 ns/op 550.00 ns/op 0.93
mainnet_e58758 - phase0 processEth1DataReset 496.00 ns/op 616.00 ns/op 0.81
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.0057 ms/op 1.0018 ms/op 1.00
mainnet_e58758 - phase0 processSlashingsReset 4.4520 us/op 3.6040 us/op 1.24
mainnet_e58758 - phase0 processRandaoMixesReset 4.6530 us/op 4.3220 us/op 1.08
mainnet_e58758 - phase0 processHistoricalRootsUpdate 545.00 ns/op 546.00 ns/op 1.00
mainnet_e58758 - phase0 processParticipationRecordUpdates 4.5780 us/op 4.0130 us/op 1.14
mainnet_e58758 - phase0 afterProcessEpoch 98.579 ms/op 99.063 ms/op 1.00
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.2311 ms/op 1.2388 ms/op 0.99
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.4872 ms/op 1.5176 ms/op 0.98
altair processInactivityUpdates - 250000 normalcase 20.448 ms/op 19.940 ms/op 1.03
altair processInactivityUpdates - 250000 worstcase 23.838 ms/op 27.064 ms/op 0.88
phase0 processRegistryUpdates - 250000 normalcase 5.8630 us/op 7.1780 us/op 0.82
phase0 processRegistryUpdates - 250000 badcase_full_deposits 263.59 us/op 269.31 us/op 0.98
phase0 processRegistryUpdates - 250000 worstcase 0.5 118.81 ms/op 114.01 ms/op 1.04
altair processRewardsAndPenalties - 250000 normalcase 53.140 ms/op 66.560 ms/op 0.80
altair processRewardsAndPenalties - 250000 worstcase 55.732 ms/op 85.659 ms/op 0.65
phase0 getAttestationDeltas - 250000 normalcase 6.6967 ms/op 13.268 ms/op 0.50
phase0 getAttestationDeltas - 250000 worstcase 6.7305 ms/op 11.472 ms/op 0.59
phase0 processSlashings - 250000 worstcase 3.5510 ms/op 5.2295 ms/op 0.68
altair processSyncCommitteeUpdates - 250000 175.36 ms/op 267.11 ms/op 0.66
BeaconState.hashTreeRoot - No change 258.00 ns/op 272.00 ns/op 0.95
BeaconState.hashTreeRoot - 1 full validator 51.599 us/op 56.044 us/op 0.92
BeaconState.hashTreeRoot - 32 full validator 536.94 us/op 555.32 us/op 0.97
BeaconState.hashTreeRoot - 512 full validator 5.2854 ms/op 5.1107 ms/op 1.03
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 66.762 us/op 64.968 us/op 1.03
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 901.55 us/op 900.49 us/op 1.00
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 12.059 ms/op 12.402 ms/op 0.97
BeaconState.hashTreeRoot - 1 balances 50.589 us/op 49.128 us/op 1.03
BeaconState.hashTreeRoot - 32 balances 446.61 us/op 483.02 us/op 0.92
BeaconState.hashTreeRoot - 512 balances 4.4766 ms/op 4.8514 ms/op 0.92
BeaconState.hashTreeRoot - 250000 balances 71.446 ms/op 77.197 ms/op 0.93
aggregationBits - 2048 els - zipIndexesInBitList 17.446 us/op 18.213 us/op 0.96
regular array get 100000 times 42.225 us/op 40.968 us/op 1.03
wrappedArray get 100000 times 32.903 us/op 33.735 us/op 0.98
arrayWithProxy get 100000 times 16.075 ms/op 15.639 ms/op 1.03
ssz.Root.equals 556.00 ns/op 611.00 ns/op 0.91
byteArrayEquals 558.00 ns/op 587.00 ns/op 0.95
shuffle list - 16384 els 6.8782 ms/op 6.8495 ms/op 1.00
shuffle list - 250000 els 101.32 ms/op 100.60 ms/op 1.01
processSlot - 1 slots 8.6640 us/op 9.1180 us/op 0.95
processSlot - 32 slots 1.3774 ms/op 1.3833 ms/op 1.00
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 199.50 us/op 204.65 us/op 0.97
getCommitteeAssignments - req 1 vs - 250000 vc 2.8882 ms/op 2.9448 ms/op 0.98
getCommitteeAssignments - req 100 vs - 250000 vc 4.1464 ms/op 4.1345 ms/op 1.00
getCommitteeAssignments - req 1000 vs - 250000 vc 4.4586 ms/op 4.4733 ms/op 1.00
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.7900 ns/op 4.9600 ns/op 0.97
state getBlockRootAtSlot - 250000 vs - 7PWei 1.0432 us/op 1.0615 us/op 0.98
computeProposers - vc 250000 10.460 ms/op 10.780 ms/op 0.97
computeEpochShuffling - vc 250000 103.78 ms/op 102.74 ms/op 1.01
getNextSyncCommittee - vc 250000 180.79 ms/op 171.52 ms/op 1.05

by benchmarkbot/action

@twoeths twoeths marked this pull request as ready for review February 2, 2023 08:04
@twoeths twoeths requested a review from a team as a code owner February 2, 2023 08:04
nflaig
nflaig reviewed Feb 2, 2023
View reviewed changes
dashboards/lodestar_debug_gossipsub.json
@@ -92,6 +45,10 @@
"panels": [
{
"collapsed": false,
"datasource": {
"type": "prometheus",
"uid": "prometheus_local"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to use ${DS_PROMETHEUS} here, you can just run node scripts/lint-grafana-dashboards.mjs ./dashboards to fix the dashboard

Suggested change
"uid": "prometheus_local"
"uid": "${DS_PROMETHEUS}"

dashboards/lodestar_debug_gossipsub.json Outdated
@@ -1,58 +1,12 @@
{
"__inputs": [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this __inputs header be removed? I think if you do Export for sharing externally it is added

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure, not related to this change so I reverted

@twoeths twoeths force-pushed the tuyen/iwant_promise_never_delivered_metric branch from 7fdda9e to ddb2568 Compare February 2, 2023 23:04
@wemeetagain wemeetagain merged commit 0efcb16 into unstable Feb 7, 2023
@wemeetagain wemeetagain deleted the tuyen/iwant_promise_never_delivered_metric branch February 7, 2023 19:18
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.6.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nflaig nflaig nflaig left review comments

@wemeetagain wemeetagain wemeetagain approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@twoeths @wemeetagain @nflaig