Provision to enable and specify batchsize for backfill sync (disabled by default)

[g11tech]

This PR provides one with the provision to enable and specify batchsize for backfill sync via providing a non zero number for sync.backfillBatchSize. By default this is zero and hence will not start any backfill sync.

This may be later in subsequent PRs set to default as 64 once the optimization issues are resolved.

[codecov]

Codecov Report

Merging #3763 (b391b06) into master (2d4c9c2) will not change coverage.
The diff coverage is n/a.

❗ Current head b391b06 differs from pull request most recent head e33e918. Consider uploading reports for the commit e33e918 to get more accurate results

@@           Coverage Diff           @@
##           master    #3763   +/-   ##
=======================================
  Coverage   36.84%   36.84%           
=======================================
  Files         320      320           
  Lines        8772     8772           
  Branches     1365     1365           
=======================================
  Hits         3232     3232           
  Misses       5398     5398           
  Partials      142      142           

[github-actions]

Performance Report

✔️ no performance regression detected

Full benchmark results

Benchmark suite	Current: b391b06	Previous: 2d4c9c2	Ratio
BeaconState.hashTreeRoot - No change	525.00 ns/op	648.00 ns/op	0.81
BeaconState.hashTreeRoot - 1 full validator	125.20 us/op	152.45 us/op	0.82
BeaconState.hashTreeRoot - 32 full validator	1.9225 ms/op	2.4181 ms/op	0.80
BeaconState.hashTreeRoot - 512 full validator	24.544 ms/op	30.614 ms/op	0.80
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	136.92 us/op	150.91 us/op	0.91
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	2.0245 ms/op	2.4542 ms/op	0.82
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	26.404 ms/op	33.763 ms/op	0.78
BeaconState.hashTreeRoot - 1 balances	89.748 us/op	110.25 us/op	0.81
BeaconState.hashTreeRoot - 32 balances	742.54 us/op	922.59 us/op	0.80
BeaconState.hashTreeRoot - 512 balances	7.2468 ms/op	8.9168 ms/op	0.81
BeaconState.hashTreeRoot - 250000 balances	142.73 ms/op	159.61 ms/op	0.89
processSlot - 1 slots	53.282 us/op	56.143 us/op	0.95
processSlot - 32 slots	2.7610 ms/op	3.4105 ms/op	0.81
getCommitteeAssignments - req 1 vs - 250000 vc	5.3185 ms/op	6.2897 ms/op	0.85
getCommitteeAssignments - req 100 vs - 250000 vc	7.3765 ms/op	8.7322 ms/op	0.84
getCommitteeAssignments - req 1000 vs - 250000 vc	7.9463 ms/op	9.3566 ms/op	0.85
computeProposers - vc 250000	19.750 ms/op	25.006 ms/op	0.79
computeEpochShuffling - vc 250000	181.84 ms/op	224.07 ms/op	0.81
getNextSyncCommittee - vc 250000	326.27 ms/op	408.46 ms/op	0.80
altair processAttestation - 250000 vs - 7PWei normalcase	34.803 ms/op	41.553 ms/op	0.84
altair processAttestation - 250000 vs - 7PWei worstcase	34.128 ms/op	39.908 ms/op	0.86
altair processAttestation - setStatus - 1/6 committees join	10.350 ms/op	13.044 ms/op	0.79
altair processAttestation - setStatus - 1/3 committees join	21.373 ms/op	25.327 ms/op	0.84
altair processAttestation - setStatus - 1/2 committees join	35.404 ms/op	42.995 ms/op	0.82
altair processAttestation - setStatus - 2/3 committees join	40.331 ms/op	57.478 ms/op	0.70
altair processAttestation - setStatus - 4/5 committees join	49.394 ms/op	72.836 ms/op	0.68
altair processAttestation - setStatus - 100% committees join	68.208 ms/op	89.826 ms/op	0.76
altair processAttestation - updateEpochParticipants - 1/6 committees join	9.8627 ms/op	14.119 ms/op	0.70
altair processAttestation - updateEpochParticipants - 1/3 committees join	21.118 ms/op	29.283 ms/op	0.72
altair processAttestation - updateEpochParticipants - 1/2 committees join	23.800 ms/op	24.601 ms/op	0.97
altair processAttestation - updateEpochParticipants - 2/3 committees join	21.844 ms/op	31.056 ms/op	0.70
altair processAttestation - updateEpochParticipants - 4/5 committees join	26.371 ms/op	26.255 ms/op	1.00
altair processAttestation - updateEpochParticipants - 100% committees join	23.480 ms/op	28.862 ms/op	0.81
altair processAttestation - updateAllStatus	18.916 ms/op	23.376 ms/op	0.81
altair processBlock - 250000 vs - 7PWei normalcase	32.638 ms/op	46.340 ms/op	0.70
altair processBlock - 250000 vs - 7PWei worstcase	109.46 ms/op	125.02 ms/op	0.88
altair processEpoch - mainnet_e81889	911.04 ms/op	1.0922 s/op	0.83
mainnet_e81889 - altair beforeProcessEpoch	266.21 ms/op	294.65 ms/op	0.90
mainnet_e81889 - altair processJustificationAndFinalization	107.03 us/op	70.865 us/op	1.51
mainnet_e81889 - altair processInactivityUpdates	17.975 ms/op	21.051 ms/op	0.85
mainnet_e81889 - altair processRewardsAndPenalties	120.53 ms/op	147.89 ms/op	0.82
mainnet_e81889 - altair processRegistryUpdates	13.042 us/op	12.631 us/op	1.03
mainnet_e81889 - altair processSlashings	2.2310 us/op	3.9020 us/op	0.57
mainnet_e81889 - altair processEth1DataReset	2.2520 us/op	2.9610 us/op	0.76
mainnet_e81889 - altair processEffectiveBalanceUpdates	6.9907 ms/op	8.3711 ms/op	0.84
mainnet_e81889 - altair processSlashingsReset	18.527 us/op	17.428 us/op	1.06
mainnet_e81889 - altair processRandaoMixesReset	24.832 us/op	22.923 us/op	1.08
mainnet_e81889 - altair processHistoricalRootsUpdate	2.7820 us/op	4.2020 us/op	0.66
mainnet_e81889 - altair processParticipationFlagUpdates	66.514 ms/op	85.173 ms/op	0.78
mainnet_e81889 - altair processSyncCommitteeUpdates	2.5040 us/op	2.7240 us/op	0.92
mainnet_e81889 - altair afterProcessEpoch	214.51 ms/op	266.39 ms/op	0.81
altair processInactivityUpdates - 250000 normalcase	65.736 ms/op	85.095 ms/op	0.77
altair processInactivityUpdates - 250000 worstcase	65.622 ms/op	83.171 ms/op	0.79
altair processParticipationFlagUpdates - 250000 anycase	59.148 ms/op	73.085 ms/op	0.81
altair processRewardsAndPenalties - 250000 normalcase	100.37 ms/op	125.32 ms/op	0.80
altair processRewardsAndPenalties - 250000 worstcase	96.473 ms/op	88.395 ms/op	1.09
altair processSyncCommitteeUpdates - 250000	363.81 ms/op	421.71 ms/op	0.86
Tree 40 250000 create	803.55 ms/op	773.53 ms/op	1.04
Tree 40 250000 get(125000)	319.15 ns/op	394.39 ns/op	0.81
Tree 40 250000 set(125000)	1.7917 us/op	2.4206 us/op	0.74
Tree 40 250000 toArray()	42.799 ms/op	47.163 ms/op	0.91
Tree 40 250000 iterate all - toArray() + loop	42.910 ms/op	47.903 ms/op	0.90
Tree 40 250000 iterate all - get(i)	115.63 ms/op	144.05 ms/op	0.80
MutableVector 250000 create	19.156 ms/op	31.800 ms/op	0.60
MutableVector 250000 get(125000)	12.997 ns/op	15.258 ns/op	0.85
MutableVector 250000 set(125000)	453.04 ns/op	569.01 ns/op	0.80
MutableVector 250000 toArray()	7.8753 ms/op	10.701 ms/op	0.74
MutableVector 250000 iterate all - toArray() + loop	9.2761 ms/op	10.570 ms/op	0.88
MutableVector 250000 iterate all - get(i)	3.3105 ms/op	4.0026 ms/op	0.83
Array 250000 create	4.5100 ms/op	7.0711 ms/op	0.64
Array 250000 clone - spread	1.6579 ms/op	2.8448 ms/op	0.58
Array 250000 get(125000)	0.85100 ns/op	1.2850 ns/op	0.66
Array 250000 set(125000)	0.84600 ns/op	1.2420 ns/op	0.68
Array 250000 iterate all - loop	167.79 us/op	202.76 us/op	0.83
effectiveBalanceIncrements clone Uint8Array 300000	51.752 us/op	85.725 us/op	0.60
effectiveBalanceIncrements clone MutableVector 300000	495.00 ns/op	794.00 ns/op	0.62
effectiveBalanceIncrements rw all Uint8Array 300000	301.72 us/op	362.23 us/op	0.83
effectiveBalanceIncrements rw all MutableVector 300000	149.57 ms/op	206.75 ms/op	0.72
aggregationBits - 2048 els - readonlyValues	171.16 us/op	222.06 us/op	0.77
aggregationBits - 2048 els - zipIndexesInBitList	32.209 us/op	40.239 us/op	0.80
regular array get 100000 times	67.375 us/op	80.943 us/op	0.83
wrappedArray get 100000 times	67.432 us/op	81.453 us/op	0.83
arrayWithProxy get 100000 times	35.164 ms/op	33.952 ms/op	1.04
ssz.Root.equals	1.2350 us/op	1.2940 us/op	0.95
ssz.Root.equals with valueOf()	1.2080 us/op	1.5870 us/op	0.76
byteArrayEquals with valueOf()	1.2770 us/op	1.5140 us/op	0.84
phase0 processBlock - 250000 vs - 7PWei normalcase	7.2302 ms/op	9.1665 ms/op	0.79
phase0 processBlock - 250000 vs - 7PWei worstcase	69.374 ms/op	90.396 ms/op	0.77
phase0 afterProcessEpoch - 250000 vs - 7PWei	201.01 ms/op	259.28 ms/op	0.78
phase0 beforeProcessEpoch - 250000 vs - 7PWei	526.48 ms/op	686.17 ms/op	0.77
phase0 processEpoch - mainnet_e58758	825.06 ms/op	995.81 ms/op	0.83
mainnet_e58758 - phase0 beforeProcessEpoch	412.93 ms/op	489.66 ms/op	0.84
mainnet_e58758 - phase0 processJustificationAndFinalization	113.89 us/op	64.683 us/op	1.76
mainnet_e58758 - phase0 processRewardsAndPenalties	109.11 ms/op	95.529 ms/op	1.14
mainnet_e58758 - phase0 processRegistryUpdates	72.679 us/op	50.148 us/op	1.45
mainnet_e58758 - phase0 processSlashings	2.3440 us/op	3.3650 us/op	0.70
mainnet_e58758 - phase0 processEth1DataReset	2.3510 us/op	2.9050 us/op	0.81
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	5.7983 ms/op	6.4847 ms/op	0.89
mainnet_e58758 - phase0 processSlashingsReset	11.823 us/op	14.482 us/op	0.82
mainnet_e58758 - phase0 processRandaoMixesReset	24.230 us/op	18.904 us/op	1.28
mainnet_e58758 - phase0 processHistoricalRootsUpdate	2.8570 us/op	3.8790 us/op	0.74
mainnet_e58758 - phase0 processParticipationRecordUpdates	17.899 us/op	14.842 us/op	1.21
mainnet_e58758 - phase0 afterProcessEpoch	176.35 ms/op	225.51 ms/op	0.78
phase0 processEffectiveBalanceUpdates - 250000 normalcase	6.3861 ms/op	7.4940 ms/op	0.85
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	6.8035 ms/op	8.4782 ms/op	0.80
phase0 processRegistryUpdates - 250000 normalcase	60.315 us/op	47.731 us/op	1.26
phase0 processRegistryUpdates - 250000 badcase_full_deposits	2.9920 ms/op	3.5892 ms/op	0.83
phase0 processRegistryUpdates - 250000 worstcase 0.5	1.6326 s/op	2.1061 s/op	0.78
phase0 getAttestationDeltas - 250000 normalcase	21.948 ms/op	15.577 ms/op	1.41
phase0 getAttestationDeltas - 250000 worstcase	13.150 ms/op	15.508 ms/op	0.85
phase0 processSlashings - 250000 worstcase	30.036 ms/op	37.426 ms/op	0.80
shuffle list - 16384 els	12.387 ms/op	16.608 ms/op	0.75
shuffle list - 250000 els	179.36 ms/op	254.15 ms/op	0.71
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	417.64 us/op	593.12 us/op	0.70
pass gossip attestations to forkchoice per slot	18.570 ms/op	18.903 ms/op	0.98
computeDeltas	3.5558 ms/op	3.7884 ms/op	0.94
computeProposerBoostScoreFromBalances	502.76 us/op	602.55 us/op	0.83
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	1.8634 ms/op	2.4624 ms/op	0.76
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	695.05 us/op	875.81 us/op	0.79
BLS verify - blst-native	1.8590 ms/op	2.2313 ms/op	0.83
BLS verifyMultipleSignatures 3 - blst-native	3.8141 ms/op	4.5838 ms/op	0.83
BLS verifyMultipleSignatures 8 - blst-native	8.2249 ms/op	9.8978 ms/op	0.83
BLS verifyMultipleSignatures 32 - blst-native	30.067 ms/op	35.873 ms/op	0.84
BLS aggregatePubkeys 32 - blst-native	39.416 us/op	47.321 us/op	0.83
BLS aggregatePubkeys 128 - blst-native	153.38 us/op	184.34 us/op	0.83
getAttestationsForBlock	58.865 ms/op	78.161 ms/op	0.75
CheckpointStateCache - add get delete	17.194 us/op	21.828 us/op	0.79
validate gossip signedAggregateAndProof - struct	4.4768 ms/op	5.6691 ms/op	0.79
validate gossip signedAggregateAndProof - treeBacked	4.3869 ms/op	5.3283 ms/op	0.82
validate gossip attestation - struct	2.0816 ms/op	2.5230 ms/op	0.83
validate gossip attestation - treeBacked	2.1059 ms/op	2.5273 ms/op	0.83
pickEth1Vote - no votes	7.9369 ms/op	10.901 ms/op	0.73
pickEth1Vote - max votes	46.721 ms/op	60.475 ms/op	0.77
pickEth1Vote - Eth1Data hashTreeRoot value x2048	23.692 ms/op	29.616 ms/op	0.80
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	9.1764 ms/op	11.638 ms/op	0.79
pickEth1Vote - Eth1Data fastSerialize value x2048	4.9584 ms/op	6.5180 ms/op	0.76
pickEth1Vote - Eth1Data fastSerialize tree x2048	23.322 ms/op	29.472 ms/op	0.79
bytes32 toHexString	1.5350 us/op	2.0950 us/op	0.73
bytes32 Buffer.toString(hex)	636.00 ns/op	849.00 ns/op	0.75
bytes32 Buffer.toString(hex) from Uint8Array	847.00 ns/op	1.0810 us/op	0.78
bytes32 Buffer.toString(hex) + 0x	642.00 ns/op	837.00 ns/op	0.77
Object access 1 prop	0.29300 ns/op	0.39500 ns/op	0.74
Map access 1 prop	0.27700 ns/op	0.34300 ns/op	0.81
Object get x1000	18.637 ns/op	22.025 ns/op	0.85
Map get x1000	0.98000 ns/op	1.1500 ns/op	0.85
Object set x1000	98.596 ns/op	132.60 ns/op	0.74
Map set x1000	57.605 ns/op	78.015 ns/op	0.74
Return object 10000 times	0.37110 ns/op	0.44450 ns/op	0.83
Throw Error 10000 times	5.9051 us/op	7.1873 us/op	0.82
enrSubnets - fastDeserialize 64 bits	1.1930 us/op	1.4830 us/op	0.80
enrSubnets - ssz BitVector 64 bits	16.253 us/op	20.413 us/op	0.80
enrSubnets - fastDeserialize 4 bits	411.00 ns/op	521.00 ns/op	0.79
enrSubnets - ssz BitVector 4 bits	2.8300 us/op	3.5170 us/op	0.80
RateTracker 1000000 limit, 1 obj count per request	180.34 ns/op	214.49 ns/op	0.84
RateTracker 1000000 limit, 2 obj count per request	129.56 ns/op	160.03 ns/op	0.81
RateTracker 1000000 limit, 4 obj count per request	105.68 ns/op	133.39 ns/op	0.79
RateTracker 1000000 limit, 8 obj count per request	95.084 ns/op	116.56 ns/op	0.82
RateTracker with prune	3.5640 us/op	4.2580 us/op	0.84
array of 16000 items push then shift	3.1232 us/op	3.7335 us/op	0.84
LinkedList of 16000 items push then shift	17.298 ns/op	21.230 ns/op	0.81
array of 16000 items push then pop	185.66 ns/op	248.82 ns/op	0.75
LinkedList of 16000 items push then pop	16.737 ns/op	19.372 ns/op	0.86
array of 24000 items push then shift	4.5495 us/op	5.4351 us/op	0.84
LinkedList of 24000 items push then shift	18.893 ns/op	22.432 ns/op	0.84
array of 24000 items push then pop	178.20 ns/op	225.06 ns/op	0.79
LinkedList of 24000 items push then pop	18.405 ns/op	20.662 ns/op	0.89

by benchmarkbot/action