Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: update the test runner #7404

Open
wants to merge 44 commits into
base: unstable
Choose a base branch
from
Open

chore: update the test runner #7404

wants to merge 44 commits into from
+1,664 −1,607

Conversation

nazarhussain
Copy link
Contributor

@nazarhussain nazarhussain commented Jan 27, 2025 •
edited
Loading

Motivation

Update vitest and browser engine.

Description

  • Move root level shared configs to configs directory
  • Update to vitest 3.x
  • Migrate from webdriverio to playwright which is more stable and recommended by Vitest.

NOTE
Reverted the playwright upgrade. See the comments.

lodestar/configs/vitest.config.base.browser.ts

Lines 41 to 43 in d2a7af0

// Recommended provider is `playwright` but it's causing following error on CI
// Error: Failed to connect to the browser session "af5be85a-7f29-4299-b680-b07f0cfc2520" within the timeout.
// TODO: Debug the issue in later versions of playwright and vitest

Steps to test or reproduce

Run all tests

@nazarhussain nazarhussain requested a review from a team as a code owner January 27, 2025 11:28
@nazarhussain nazarhussain marked this pull request as draft January 27, 2025 11:28
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 27, 2025 •
edited
Loading

⚠️ Performance Alert ⚠️

Possible performance regression was detected for some benchmarks.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold.

Benchmark suite Current: 31409da Previous: bb5bf12 Ratio
forkChoice updateHead vc 600000 bc 64 eq 300000 71.535 ms/op 17.054 ms/op 4.19
Full benchmark results
Benchmark suite Current: 31409da Previous: bb5bf12 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 1.0929 ms/op 1.9048 ms/op 0.57
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 39.031 us/op 74.215 us/op 0.53
BLS verify - blst 1.0875 ms/op 1.4105 ms/op 0.77
BLS verifyMultipleSignatures 3 - blst 1.3140 ms/op 2.0798 ms/op 0.63
BLS verifyMultipleSignatures 8 - blst 1.9414 ms/op 2.8922 ms/op 0.67
BLS verifyMultipleSignatures 32 - blst 6.0262 ms/op 8.4336 ms/op 0.71
BLS verifyMultipleSignatures 64 - blst 11.498 ms/op 14.999 ms/op 0.77
BLS verifyMultipleSignatures 128 - blst 18.354 ms/op 30.325 ms/op 0.61
BLS deserializing 10000 signatures 709.20 ms/op 1.0322 s/op 0.69
BLS deserializing 100000 signatures 7.2958 s/op 9.5444 s/op 0.76
BLS verifyMultipleSignatures - same message - 3 - blst 1.0441 ms/op 1.1305 ms/op 0.92
BLS verifyMultipleSignatures - same message - 8 - blst 1.2492 ms/op 1.4157 ms/op 0.88
BLS verifyMultipleSignatures - same message - 32 - blst 1.9260 ms/op 2.6027 ms/op 0.74
BLS verifyMultipleSignatures - same message - 64 - blst 2.8095 ms/op 3.0955 ms/op 0.91
BLS verifyMultipleSignatures - same message - 128 - blst 4.7798 ms/op 5.4344 ms/op 0.88
BLS aggregatePubkeys 32 - blst 20.631 us/op 24.593 us/op 0.84
BLS aggregatePubkeys 128 - blst 73.166 us/op 91.110 us/op 0.80
notSeenSlots=1 numMissedVotes=1 numBadVotes=10 61.861 ms/op 61.830 ms/op 1.00
notSeenSlots=1 numMissedVotes=0 numBadVotes=4 60.163 ms/op 59.398 ms/op 1.01
notSeenSlots=2 numMissedVotes=1 numBadVotes=10 49.198 ms/op 47.297 ms/op 1.04
getSlashingsAndExits - default max 116.84 us/op 102.57 us/op 1.14
getSlashingsAndExits - 2k 483.82 us/op 354.75 us/op 1.36
proposeBlockBody type=full, size=empty 6.1198 ms/op 5.3163 ms/op 1.15
isKnown best case - 1 super set check 223.00 ns/op 227.00 ns/op 0.98
isKnown normal case - 2 super set checks 259.00 ns/op 222.00 ns/op 1.17
isKnown worse case - 16 super set checks 217.00 ns/op 220.00 ns/op 0.99
InMemoryCheckpointStateCache - add get delete 3.3000 us/op 2.9170 us/op 1.13
validate api signedAggregateAndProof - struct 1.9121 ms/op 1.6129 ms/op 1.19
validate gossip signedAggregateAndProof - struct 1.6068 ms/op 1.8625 ms/op 0.86
batch validate gossip attestation - vc 640000 - chunk 32 171.19 us/op 149.86 us/op 1.14
batch validate gossip attestation - vc 640000 - chunk 64 158.93 us/op 118.86 us/op 1.34
batch validate gossip attestation - vc 640000 - chunk 128 145.01 us/op 105.29 us/op 1.38
batch validate gossip attestation - vc 640000 - chunk 256 144.75 us/op 119.84 us/op 1.21
pickEth1Vote - no votes 1.1951 ms/op 1.1691 ms/op 1.02
pickEth1Vote - max votes 10.086 ms/op 8.5281 ms/op 1.18
pickEth1Vote - Eth1Data hashTreeRoot value x2048 20.018 ms/op 14.251 ms/op 1.40
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 32.095 ms/op 19.308 ms/op 1.66
pickEth1Vote - Eth1Data fastSerialize value x2048 479.79 us/op 519.68 us/op 0.92
pickEth1Vote - Eth1Data fastSerialize tree x2048 4.9722 ms/op 2.4392 ms/op 2.04
bytes32 toHexString 397.00 ns/op 368.00 ns/op 1.08
bytes32 Buffer.toString(hex) 530.00 ns/op 263.00 ns/op 2.02
bytes32 Buffer.toString(hex) from Uint8Array 754.00 ns/op 401.00 ns/op 1.88
bytes32 Buffer.toString(hex) + 0x 384.00 ns/op 279.00 ns/op 1.38
Object access 1 prop 0.16800 ns/op 0.12000 ns/op 1.40
Map access 1 prop 0.14100 ns/op 0.13800 ns/op 1.02
Object get x1000 6.5270 ns/op 6.5810 ns/op 0.99
Map get x1000 6.8760 ns/op 6.7030 ns/op 1.03
Object set x1000 34.925 ns/op 29.740 ns/op 1.17
Map set x1000 23.572 ns/op 21.031 ns/op 1.12
Return object 10000 times 0.31650 ns/op 0.31490 ns/op 1.01
Throw Error 10000 times 5.5609 us/op 5.3589 us/op 1.04
toHex 151.11 ns/op 176.40 ns/op 0.86
Buffer.from 140.58 ns/op 142.35 ns/op 0.99
shared Buffer 98.699 ns/op 95.852 ns/op 1.03
fastMsgIdFn sha256 / 200 bytes 2.5130 us/op 2.2510 us/op 1.12
fastMsgIdFn h32 xxhash / 200 bytes 229.00 ns/op 210.00 ns/op 1.09
fastMsgIdFn h64 xxhash / 200 bytes 320.00 ns/op 277.00 ns/op 1.16
fastMsgIdFn sha256 / 1000 bytes 8.7140 us/op 7.3520 us/op 1.19
fastMsgIdFn h32 xxhash / 1000 bytes 366.00 ns/op 360.00 ns/op 1.02
fastMsgIdFn h64 xxhash / 1000 bytes 472.00 ns/op 389.00 ns/op 1.21
fastMsgIdFn sha256 / 10000 bytes 94.580 us/op 73.791 us/op 1.28
fastMsgIdFn h32 xxhash / 10000 bytes 2.0420 us/op 1.8170 us/op 1.12
fastMsgIdFn h64 xxhash / 10000 bytes 1.6350 us/op 1.2350 us/op 1.32
send data - 1000 256B messages 21.501 ms/op 15.345 ms/op 1.40
send data - 1000 512B messages 24.966 ms/op 21.602 ms/op 1.16
send data - 1000 1024B messages 33.360 ms/op 30.119 ms/op 1.11
send data - 1000 1200B messages 29.339 ms/op 25.355 ms/op 1.16
send data - 1000 2048B messages 36.364 ms/op 27.996 ms/op 1.30
send data - 1000 4096B messages 37.090 ms/op 35.343 ms/op 1.05
send data - 1000 16384B messages 102.16 ms/op 79.844 ms/op 1.28
send data - 1000 65536B messages 675.73 ms/op 234.77 ms/op 2.88
enrSubnets - fastDeserialize 64 bits 1.2900 us/op 916.00 ns/op 1.41
enrSubnets - ssz BitVector 64 bits 374.00 ns/op 330.00 ns/op 1.13
enrSubnets - fastDeserialize 4 bits 172.00 ns/op 133.00 ns/op 1.29
enrSubnets - ssz BitVector 4 bits 476.00 ns/op 334.00 ns/op 1.43
prioritizePeers score -10:0 att 32-0.1 sync 2-0 169.01 us/op 121.38 us/op 1.39
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 311.44 us/op 142.79 us/op 2.18
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 338.38 us/op 206.27 us/op 1.64
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 550.92 us/op 436.63 us/op 1.26
prioritizePeers score 0:0 att 64-1 sync 4-1 614.60 us/op 527.32 us/op 1.17
array of 16000 items push then shift 1.8839 us/op 1.8264 us/op 1.03
LinkedList of 16000 items push then shift 15.332 ns/op 7.2250 ns/op 2.12
array of 16000 items push then pop 136.41 ns/op 76.405 ns/op 1.79
LinkedList of 16000 items push then pop 12.709 ns/op 6.9910 ns/op 1.82
array of 24000 items push then shift 3.5326 us/op 2.4153 us/op 1.46
LinkedList of 24000 items push then shift 10.046 ns/op 7.1100 ns/op 1.41
array of 24000 items push then pop 206.11 ns/op 101.13 ns/op 2.04
LinkedList of 24000 items push then pop 8.5490 ns/op 6.9590 ns/op 1.23
intersect bitArray bitLen 8 7.1240 ns/op 6.4750 ns/op 1.10
intersect array and set length 8 61.716 ns/op 38.361 ns/op 1.61
intersect bitArray bitLen 128 53.691 ns/op 30.294 ns/op 1.77
intersect array and set length 128 1.0994 us/op 637.19 ns/op 1.73
bitArray.getTrueBitIndexes() bitLen 128 1.3350 us/op 1.0270 us/op 1.30
bitArray.getTrueBitIndexes() bitLen 248 2.0770 us/op 1.8000 us/op 1.15
bitArray.getTrueBitIndexes() bitLen 512 4.4980 us/op 3.6730 us/op 1.22
Buffer.concat 32 items 762.00 ns/op 810.00 ns/op 0.94
Uint8Array.set 32 items 2.3050 us/op 1.5520 us/op 1.49
Buffer.copy 4.1570 us/op 3.7020 us/op 1.12
Uint8Array.set - with subarray 3.4130 us/op 2.8610 us/op 1.19
Uint8Array.set - without subarray 2.1830 us/op 1.7000 us/op 1.28
getUint32 - dataview 221.00 ns/op 226.00 ns/op 0.98
getUint32 - manual 131.00 ns/op 135.00 ns/op 0.97
Set add up to 64 items then delete first 3.3338 us/op 2.4734 us/op 1.35
OrderedSet add up to 64 items then delete first 5.4663 us/op 3.7421 us/op 1.46
Set add up to 64 items then delete last 3.8772 us/op 2.9691 us/op 1.31
OrderedSet add up to 64 items then delete last 5.4431 us/op 4.6679 us/op 1.17
Set add up to 64 items then delete middle 3.8202 us/op 2.9747 us/op 1.28
OrderedSet add up to 64 items then delete middle 7.8679 us/op 5.7002 us/op 1.38
Set add up to 128 items then delete first 7.1824 us/op 5.7020 us/op 1.26
OrderedSet add up to 128 items then delete first 17.627 us/op 9.9212 us/op 1.78
Set add up to 128 items then delete last 7.3393 us/op 6.6456 us/op 1.10
OrderedSet add up to 128 items then delete last 11.236 us/op 7.9602 us/op 1.41
Set add up to 128 items then delete middle 9.4267 us/op 5.0389 us/op 1.87
OrderedSet add up to 128 items then delete middle 26.293 us/op 14.699 us/op 1.79
Set add up to 256 items then delete first 15.266 us/op 11.208 us/op 1.36
OrderedSet add up to 256 items then delete first 24.556 us/op 16.934 us/op 1.45
Set add up to 256 items then delete last 18.372 us/op 10.519 us/op 1.75
OrderedSet add up to 256 items then delete last 29.254 us/op 16.226 us/op 1.80
Set add up to 256 items then delete middle 20.814 us/op 11.778 us/op 1.77
OrderedSet add up to 256 items then delete middle 65.485 us/op 45.786 us/op 1.43
transfer serialized Status (84 B) 5.6180 us/op 2.7330 us/op 2.06
copy serialized Status (84 B) 3.8130 us/op 2.0000 us/op 1.91
transfer serialized SignedVoluntaryExit (112 B) 5.7590 us/op 2.8330 us/op 2.03
copy serialized SignedVoluntaryExit (112 B) 3.0110 us/op 1.6310 us/op 1.85
transfer serialized ProposerSlashing (416 B) 5.2810 us/op 2.8840 us/op 1.83
copy serialized ProposerSlashing (416 B) 3.3380 us/op 2.3830 us/op 1.40
transfer serialized Attestation (485 B) 6.3940 us/op 3.5220 us/op 1.82
copy serialized Attestation (485 B) 3.9970 us/op 2.5310 us/op 1.58
transfer serialized AttesterSlashing (33232 B) 4.6890 us/op 3.3150 us/op 1.41
copy serialized AttesterSlashing (33232 B) 9.6730 us/op 4.7750 us/op 2.03
transfer serialized Small SignedBeaconBlock (128000 B) 5.2220 us/op 3.8730 us/op 1.35
copy serialized Small SignedBeaconBlock (128000 B) 19.598 us/op 11.587 us/op 1.69
transfer serialized Avg SignedBeaconBlock (200000 B) 9.3480 us/op 4.0270 us/op 2.32
copy serialized Avg SignedBeaconBlock (200000 B) 27.447 us/op 22.233 us/op 1.23
transfer serialized BlobsSidecar (524380 B) 7.6090 us/op 4.5730 us/op 1.66
copy serialized BlobsSidecar (524380 B) 90.650 us/op 106.37 us/op 0.85
transfer serialized Big SignedBeaconBlock (1000000 B) 7.0030 us/op 4.8280 us/op 1.45
copy serialized Big SignedBeaconBlock (1000000 B) 149.30 us/op 121.13 us/op 1.23
pass gossip attestations to forkchoice per slot 3.3640 ms/op 3.0590 ms/op 1.10
forkChoice updateHead vc 100000 bc 64 eq 0 550.19 us/op 496.34 us/op 1.11
forkChoice updateHead vc 600000 bc 64 eq 0 5.0259 ms/op 3.5051 ms/op 1.43
forkChoice updateHead vc 1000000 bc 64 eq 0 6.6737 ms/op 5.4226 ms/op 1.23
forkChoice updateHead vc 600000 bc 320 eq 0 4.3667 ms/op 3.4464 ms/op 1.27
forkChoice updateHead vc 600000 bc 1200 eq 0 4.6503 ms/op 3.1893 ms/op 1.46
forkChoice updateHead vc 600000 bc 7200 eq 0 4.5195 ms/op 3.5367 ms/op 1.28
forkChoice updateHead vc 600000 bc 64 eq 1000 11.687 ms/op 12.256 ms/op 0.95
forkChoice updateHead vc 600000 bc 64 eq 10000 11.651 ms/op 12.566 ms/op 0.93
forkChoice updateHead vc 600000 bc 64 eq 300000 71.535 ms/op 17.054 ms/op 4.19
computeDeltas 500000 validators 300 proto nodes 5.6618 ms/op 4.3560 ms/op 1.30
computeDeltas 500000 validators 1200 proto nodes 6.1489 ms/op 4.4835 ms/op 1.37
computeDeltas 500000 validators 7200 proto nodes 6.1799 ms/op 4.5734 ms/op 1.35
computeDeltas 750000 validators 300 proto nodes 9.2208 ms/op 6.9005 ms/op 1.34
computeDeltas 750000 validators 1200 proto nodes 8.4989 ms/op 8.0737 ms/op 1.05
computeDeltas 750000 validators 7200 proto nodes 7.3292 ms/op 10.001 ms/op 0.73
computeDeltas 1400000 validators 300 proto nodes 13.905 ms/op 15.439 ms/op 0.90
computeDeltas 1400000 validators 1200 proto nodes 12.869 ms/op 17.006 ms/op 0.76
computeDeltas 1400000 validators 7200 proto nodes 12.075 ms/op 15.379 ms/op 0.79
computeDeltas 2100000 validators 300 proto nodes 22.367 ms/op 24.156 ms/op 0.93
computeDeltas 2100000 validators 1200 proto nodes 23.764 ms/op 22.956 ms/op 1.04
computeDeltas 2100000 validators 7200 proto nodes 19.344 ms/op 22.264 ms/op 0.87
altair processAttestation - 250000 vs - 7PWei normalcase 3.5723 ms/op 3.6731 ms/op 0.97
altair processAttestation - 250000 vs - 7PWei worstcase 4.0212 ms/op 4.9708 ms/op 0.81
altair processAttestation - setStatus - 1/6 committees join 146.56 us/op 208.06 us/op 0.70
altair processAttestation - setStatus - 1/3 committees join 290.54 us/op 306.25 us/op 0.95
altair processAttestation - setStatus - 1/2 committees join 376.62 us/op 448.73 us/op 0.84
altair processAttestation - setStatus - 2/3 committees join 475.61 us/op 718.74 us/op 0.66
altair processAttestation - setStatus - 4/5 committees join 676.60 us/op 749.12 us/op 0.90
altair processAttestation - setStatus - 100% committees join 928.86 us/op 952.70 us/op 0.97
altair processBlock - 250000 vs - 7PWei normalcase 12.075 ms/op 6.2034 ms/op 1.95
altair processBlock - 250000 vs - 7PWei normalcase hashState 41.299 ms/op 45.321 ms/op 0.91
altair processBlock - 250000 vs - 7PWei worstcase 53.262 ms/op 59.055 ms/op 0.90
altair processBlock - 250000 vs - 7PWei worstcase hashState 110.66 ms/op 113.64 ms/op 0.97
phase0 processBlock - 250000 vs - 7PWei normalcase 2.3721 ms/op 2.8754 ms/op 0.82
phase0 processBlock - 250000 vs - 7PWei worstcase 27.805 ms/op 31.597 ms/op 0.88
altair processEth1Data - 250000 vs - 7PWei normalcase 387.24 us/op 416.97 us/op 0.93
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 5.5130 us/op 8.8510 us/op 0.62
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 40.751 us/op 58.050 us/op 0.70
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 10.663 us/op 9.3340 us/op 1.14
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 6.2750 us/op 5.7570 us/op 1.09
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 163.06 us/op 133.43 us/op 1.22
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 1.2289 ms/op 1.4338 ms/op 0.86
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 1.5944 ms/op 1.7981 ms/op 0.89
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 2.2694 ms/op 1.9751 ms/op 1.15
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 10.346 ms/op 4.7318 ms/op 2.19
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 1.5448 ms/op 2.0504 ms/op 0.75
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 4.6382 ms/op 4.7669 ms/op 0.97
Tree 40 250000 create 628.22 ms/op 721.72 ms/op 0.87
Tree 40 250000 get(125000) 155.31 ns/op 187.24 ns/op 0.83
Tree 40 250000 set(125000) 2.3620 us/op 2.2271 us/op 1.06
Tree 40 250000 toArray() 23.607 ms/op 24.245 ms/op 0.97
Tree 40 250000 iterate all - toArray() + loop 23.550 ms/op 23.811 ms/op 0.99
Tree 40 250000 iterate all - get(i) 72.043 ms/op 67.120 ms/op 1.07
Array 250000 create 3.1291 ms/op 3.2422 ms/op 0.97
Array 250000 clone - spread 1.6288 ms/op 1.9112 ms/op 0.85
Array 250000 get(125000) 0.43400 ns/op 0.51100 ns/op 0.85
Array 250000 set(125000) 0.44800 ns/op 0.51600 ns/op 0.87
Array 250000 iterate all - loop 102.40 us/op 122.23 us/op 0.84
phase0 afterProcessEpoch - 250000 vs - 7PWei 53.404 ms/op 62.405 ms/op 0.86
Array.fill - length 1000000 3.9508 ms/op 5.6970 ms/op 0.69
Array push - length 1000000 21.540 ms/op 18.429 ms/op 1.17
Array.get 0.28933 ns/op 0.33832 ns/op 0.86
Uint8Array.get 0.46915 ns/op 0.64794 ns/op 0.72
phase0 beforeProcessEpoch - 250000 vs - 7PWei 24.779 ms/op 23.181 ms/op 1.07
altair processEpoch - mainnet_e81889 369.74 ms/op 342.94 ms/op 1.08
mainnet_e81889 - altair beforeProcessEpoch 26.270 ms/op 26.383 ms/op 1.00
mainnet_e81889 - altair processJustificationAndFinalization 6.1610 us/op 6.6830 us/op 0.92
mainnet_e81889 - altair processInactivityUpdates 5.7324 ms/op 5.9410 ms/op 0.96
mainnet_e81889 - altair processRewardsAndPenalties 52.923 ms/op 49.828 ms/op 1.06
mainnet_e81889 - altair processRegistryUpdates 957.00 ns/op 880.00 ns/op 1.09
mainnet_e81889 - altair processSlashings 202.00 ns/op 228.00 ns/op 0.89
mainnet_e81889 - altair processEth1DataReset 273.00 ns/op 206.00 ns/op 1.33
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.3254 ms/op 1.4278 ms/op 0.93
mainnet_e81889 - altair processSlashingsReset 1.1240 us/op 1.2700 us/op 0.89
mainnet_e81889 - altair processRandaoMixesReset 1.7320 us/op 1.5550 us/op 1.11
mainnet_e81889 - altair processHistoricalRootsUpdate 219.00 ns/op 216.00 ns/op 1.01
mainnet_e81889 - altair processParticipationFlagUpdates 748.00 ns/op 625.00 ns/op 1.20
mainnet_e81889 - altair processSyncCommitteeUpdates 169.00 ns/op 173.00 ns/op 0.98
mainnet_e81889 - altair afterProcessEpoch 57.519 ms/op 62.372 ms/op 0.92
capella processEpoch - mainnet_e217614 1.1478 s/op 1.3359 s/op 0.86
mainnet_e217614 - capella beforeProcessEpoch 76.271 ms/op 113.60 ms/op 0.67
mainnet_e217614 - capella processJustificationAndFinalization 5.8330 us/op 7.4560 us/op 0.78
mainnet_e217614 - capella processInactivityUpdates 18.184 ms/op 26.439 ms/op 0.69
mainnet_e217614 - capella processRewardsAndPenalties 210.92 ms/op 280.19 ms/op 0.75
mainnet_e217614 - capella processRegistryUpdates 6.8530 us/op 8.9510 us/op 0.77
mainnet_e217614 - capella processSlashings 246.00 ns/op 247.00 ns/op 1.00
mainnet_e217614 - capella processEth1DataReset 197.00 ns/op 306.00 ns/op 0.64
mainnet_e217614 - capella processEffectiveBalanceUpdates 15.691 ms/op 18.208 ms/op 0.86
mainnet_e217614 - capella processSlashingsReset 1.3810 us/op 1.3240 us/op 1.04
mainnet_e217614 - capella processRandaoMixesReset 1.3590 us/op 1.7000 us/op 0.80
mainnet_e217614 - capella processHistoricalRootsUpdate 198.00 ns/op 222.00 ns/op 0.89
mainnet_e217614 - capella processParticipationFlagUpdates 658.00 ns/op 699.00 ns/op 0.94
mainnet_e217614 - capella afterProcessEpoch 130.41 ms/op 150.21 ms/op 0.87
phase0 processEpoch - mainnet_e58758 371.44 ms/op 438.35 ms/op 0.85
mainnet_e58758 - phase0 beforeProcessEpoch 103.44 ms/op 120.33 ms/op 0.86
mainnet_e58758 - phase0 processJustificationAndFinalization 7.6620 us/op 11.561 us/op 0.66
mainnet_e58758 - phase0 processRewardsAndPenalties 40.122 ms/op 46.554 ms/op 0.86
mainnet_e58758 - phase0 processRegistryUpdates 3.6260 us/op 3.5640 us/op 1.02
mainnet_e58758 - phase0 processSlashings 194.00 ns/op 306.00 ns/op 0.63
mainnet_e58758 - phase0 processEth1DataReset 266.00 ns/op 217.00 ns/op 1.23
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.1268 ms/op 1.6703 ms/op 0.67
mainnet_e58758 - phase0 processSlashingsReset 1.4270 us/op 1.8350 us/op 0.78
mainnet_e58758 - phase0 processRandaoMixesReset 2.0450 us/op 3.2120 us/op 0.64
mainnet_e58758 - phase0 processHistoricalRootsUpdate 202.00 ns/op 524.00 ns/op 0.39
mainnet_e58758 - phase0 processParticipationRecordUpdates 1.0460 us/op 1.3850 us/op 0.76
mainnet_e58758 - phase0 afterProcessEpoch 45.202 ms/op 55.268 ms/op 0.82
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.5919 ms/op 1.7304 ms/op 0.92
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.9311 ms/op 3.6483 ms/op 0.80
altair processInactivityUpdates - 250000 normalcase 23.777 ms/op 24.780 ms/op 0.96
altair processInactivityUpdates - 250000 worstcase 22.957 ms/op 25.549 ms/op 0.90
phase0 processRegistryUpdates - 250000 normalcase 14.580 us/op 9.8920 us/op 1.47
phase0 processRegistryUpdates - 250000 badcase_full_deposits 444.52 us/op 335.29 us/op 1.33
phase0 processRegistryUpdates - 250000 worstcase 0.5 143.55 ms/op 157.65 ms/op 0.91
altair processRewardsAndPenalties - 250000 normalcase 62.374 ms/op 53.751 ms/op 1.16
altair processRewardsAndPenalties - 250000 worstcase 60.867 ms/op 62.643 ms/op 0.97
phase0 getAttestationDeltas - 250000 normalcase 12.433 ms/op 11.897 ms/op 1.05
phase0 getAttestationDeltas - 250000 worstcase 8.9181 ms/op 12.547 ms/op 0.71
phase0 processSlashings - 250000 worstcase 117.78 us/op 124.39 us/op 0.95
altair processSyncCommitteeUpdates - 250000 202.78 ms/op 163.58 ms/op 1.24
BeaconState.hashTreeRoot - No change 392.00 ns/op 295.00 ns/op 1.33
BeaconState.hashTreeRoot - 1 full validator 108.97 us/op 117.66 us/op 0.93
BeaconState.hashTreeRoot - 32 full validator 1.7238 ms/op 1.3476 ms/op 1.28
BeaconState.hashTreeRoot - 512 full validator 17.313 ms/op 13.053 ms/op 1.33
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 161.13 us/op 148.76 us/op 1.08
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 2.9940 ms/op 2.2932 ms/op 1.31
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 34.460 ms/op 30.512 ms/op 1.13
BeaconState.hashTreeRoot - 1 balances 104.32 us/op 116.46 us/op 0.90
BeaconState.hashTreeRoot - 32 balances 1.1721 ms/op 1.1653 ms/op 1.01
BeaconState.hashTreeRoot - 512 balances 12.119 ms/op 11.061 ms/op 1.10
BeaconState.hashTreeRoot - 250000 balances 283.35 ms/op 220.33 ms/op 1.29
aggregationBits - 2048 els - zipIndexesInBitList 31.944 us/op 27.425 us/op 1.16
byteArrayEquals 32 58.953 ns/op 72.660 ns/op 0.81
Buffer.compare 32 34.447 ns/op 21.022 ns/op 1.64
byteArrayEquals 1024 1.7571 us/op 1.9561 us/op 0.90
Buffer.compare 1024 35.779 ns/op 29.710 ns/op 1.20
byteArrayEquals 16384 31.077 us/op 30.163 us/op 1.03
Buffer.compare 16384 226.71 ns/op 243.52 ns/op 0.93
byteArrayEquals 123687377 230.77 ms/op 234.30 ms/op 0.98
Buffer.compare 123687377 18.339 ms/op 12.403 ms/op 1.48
byteArrayEquals 32 - diff last byte 58.378 ns/op 64.556 ns/op 0.90
Buffer.compare 32 - diff last byte 19.155 ns/op 20.669 ns/op 0.93
byteArrayEquals 1024 - diff last byte 1.8890 us/op 1.9950 us/op 0.95
Buffer.compare 1024 - diff last byte 32.029 ns/op 33.750 ns/op 0.95
byteArrayEquals 16384 - diff last byte 28.084 us/op 32.453 us/op 0.87
Buffer.compare 16384 - diff last byte 219.33 ns/op 244.81 ns/op 0.90
byteArrayEquals 123687377 - diff last byte 233.37 ms/op 228.23 ms/op 1.02
Buffer.compare 123687377 - diff last byte 19.780 ms/op 11.099 ms/op 1.78
byteArrayEquals 32 - random bytes 9.8650 ns/op 5.6220 ns/op 1.75
Buffer.compare 32 - random bytes 19.267 ns/op 19.213 ns/op 1.00
byteArrayEquals 1024 - random bytes 5.6260 ns/op 5.8370 ns/op 0.96
Buffer.compare 1024 - random bytes 18.949 ns/op 19.481 ns/op 0.97
byteArrayEquals 16384 - random bytes 9.6440 ns/op 5.7770 ns/op 1.67
Buffer.compare 16384 - random bytes 31.400 ns/op 19.170 ns/op 1.64
byteArrayEquals 123687377 - random bytes 8.9600 ns/op 7.6300 ns/op 1.17
Buffer.compare 123687377 - random bytes 36.010 ns/op 20.820 ns/op 1.73
regular array get 100000 times 44.920 us/op 51.281 us/op 0.88
wrappedArray get 100000 times 74.505 us/op 35.802 us/op 2.08
arrayWithProxy get 100000 times 26.519 ms/op 13.541 ms/op 1.96
ssz.Root.equals 81.571 ns/op 52.544 ns/op 1.55
byteArrayEquals 73.452 ns/op 51.121 ns/op 1.44
Buffer.compare 11.502 ns/op 11.962 ns/op 0.96
processSlot - 1 slots 22.015 us/op 13.014 us/op 1.69
processSlot - 32 slots 3.2907 ms/op 2.9419 ms/op 1.12
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 77.098 ms/op 62.725 ms/op 1.23
getCommitteeAssignments - req 1 vs - 250000 vc 2.3522 ms/op 2.6609 ms/op 0.88
getCommitteeAssignments - req 100 vs - 250000 vc 4.5173 ms/op 5.1397 ms/op 0.88
getCommitteeAssignments - req 1000 vs - 250000 vc 4.9808 ms/op 5.0681 ms/op 0.98
findModifiedValidators - 10000 modified validators 1.1453 s/op 831.65 ms/op 1.38
findModifiedValidators - 1000 modified validators 913.75 ms/op 808.64 ms/op 1.13
findModifiedValidators - 100 modified validators 363.87 ms/op 332.66 ms/op 1.09
findModifiedValidators - 10 modified validators 179.42 ms/op 155.01 ms/op 1.16
findModifiedValidators - 1 modified validators 251.32 ms/op 279.26 ms/op 0.90
findModifiedValidators - no difference 220.63 ms/op 210.30 ms/op 1.05
compare ViewDUs 8.2426 s/op 7.8692 s/op 1.05
compare each validator Uint8Array 2.6786 s/op 2.5741 s/op 1.04
compare ViewDU to Uint8Array 2.6188 s/op 2.2202 s/op 1.18
migrate state 1000000 validators, 24 modified, 0 new 1.1585 s/op 1.8873 s/op 0.61
migrate state 1000000 validators, 1700 modified, 1000 new 1.4807 s/op 2.5810 s/op 0.57
migrate state 1000000 validators, 3400 modified, 2000 new 1.6336 s/op 1.7820 s/op 0.92
migrate state 1500000 validators, 24 modified, 0 new 1.0585 s/op 1.2218 s/op 0.87
migrate state 1500000 validators, 1700 modified, 1000 new 1.2487 s/op 1.4115 s/op 0.88
migrate state 1500000 validators, 3400 modified, 2000 new 1.5695 s/op 1.6568 s/op 0.95
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.3100 ns/op 5.8800 ns/op 0.73
state getBlockRootAtSlot - 250000 vs - 7PWei 601.03 ns/op 667.56 ns/op 0.90
naive computeProposerIndex 100000 validators 70.170 ms/op 70.200 ms/op 1.00
computeProposerIndex 100000 validators 9.4559 ms/op 11.547 ms/op 0.82
naiveGetNextSyncCommitteeIndices 1000 validators 9.7191 s/op 8.9934 s/op 1.08
getNextSyncCommitteeIndices 1000 validators 267.16 ms/op 296.97 ms/op 0.90
naiveGetNextSyncCommitteeIndices 10000 validators 7.7023 s/op 9.3006 s/op 0.83
getNextSyncCommitteeIndices 10000 validators 235.14 ms/op 301.90 ms/op 0.78
naiveGetNextSyncCommitteeIndices 100000 validators 8.7354 s/op 9.6647 s/op 0.90
getNextSyncCommitteeIndices 100000 validators 346.73 ms/op 344.94 ms/op 1.01
naive computeShuffledIndex 100000 validators 27.765 s/op 29.858 s/op 0.93
cached computeShuffledIndex 100000 validators 566.20 ms/op 773.63 ms/op 0.73
naive computeShuffledIndex 2000000 validators 603.34 s/op 581.55 s/op 1.04
cached computeShuffledIndex 2000000 validators 88.985 s/op 54.797 s/op 1.62
computeProposers - vc 250000 21.799 ms/op 10.037 ms/op 2.17
computeEpochShuffling - vc 250000 45.514 ms/op 51.011 ms/op 0.89
getNextSyncCommittee - vc 250000 230.26 ms/op 214.15 ms/op 1.08
computeSigningRoot for AttestationData 41.203 us/op 21.255 us/op 1.94
hash AttestationData serialized data then Buffer.toString(base64) 1.7859 us/op 1.6922 us/op 1.06
toHexString serialized data 1.8955 us/op 1.2135 us/op 1.56
Buffer.toString(base64) 172.13 ns/op 185.46 ns/op 0.93
nodejs block root to RootHex using toHex 161.81 ns/op 156.96 ns/op 1.03
nodejs block root to RootHex using toRootHex 93.372 ns/op 98.218 ns/op 0.95
browser block root to RootHex using the deprecated toHexString 252.54 ns/op 240.40 ns/op 1.05
browser block root to RootHex using toHex 190.73 ns/op 201.72 ns/op 0.95
browser block root to RootHex using toRootHex 169.41 ns/op 187.48 ns/op 0.90

by benchmarkbot/action

@nazarhussain nazarhussain changed the title chore: update the test runner and tests tasks chore: update the test runner and browser engine Jan 28, 2025
@nazarhussain nazarhussain marked this pull request as ready for review January 28, 2025 15:14
@wemeetagain
Copy link
Member

does your vitest-in-process-pool work with vitest 3.x?

@nazarhussain
Copy link
Contributor Author

does your vitest-in-process-pool work with vitest 3.x?

No there is some issue that is causing it to not be compatible. Was looking into it ChainSafe/benchmark#29. Will upgrade it very soon.

@codecov Codecov
Copy link

codecov bot commented Jan 29, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 48.24%. Comparing base (ee9d4fa) to head (01c80ca).
Report is 4 commits behind head on unstable.

Additional details and impacted files 
@@             Coverage Diff              @@
##           unstable    #7404      +/-   ##
============================================
- Coverage     48.45%   48.24%   -0.21%     
============================================
  Files           602      602              
  Lines         40387    40248     -139     
  Branches       2069     2069              
============================================
- Hits          19568    19417     -151     
- Misses        20781    20793      +12     
  Partials         38       38

@nazarhussain nazarhussain changed the title chore: update the test runner and browser engine chore: update the test runner Jan 30, 2025
@wemeetagain
Copy link
Member

does this PR need vitest 3.x? can we maintain 2.x for now?

@nazarhussain
Copy link
Contributor Author

does this PR need vitest 3.x? can we maintain 2.x for now?

The Vitest 3 comes with a lot of changes, but for our interests are:

Vite 6 is considered more performant (specially for frontend stack). I was more interested in workspaces and started this PR because of it, but could not get that fit in our workflow. One issue running tests from individual package folder.

So nothing critical for us, so can hold this PR merge or merge it. As you feel better.

@nazarhussain
Increase browser connection timeout
bba0a1b
@nazarhussain
Clear mocks
be93322
@nazarhussain
Update the browser configuration
78cf4b0
@nazarhussain
Update the browser config
6e69911
@nazarhussain
Fix an exception for spec tests
4bebb7e
@nazarhussain
Increase timeout
904f7c0
@nazarhussain
Increase timeout for failing test
5cf70c8
@nazarhussain
Fix lint errors
00ff38b
@nazarhussain
Add debug varible to inspect the failing browser tests
b10ad3a
@nazarhussain
Add single concurency for browser tests
08cec5b
@nazarhussain
Rename config files
281558f
@nazarhussain
Increase the timeout
1312edc
@nazarhussain
Remove unused tasks
ca2d40d
@nazarhussain
Update the browser config
fc56d59
@nazarhussain
Update the browser config
1e1eba8
@nazarhussain
Update the timeout to 15 minutes
6ea620d
@nazarhussain
Slow down playwright browser
1700ea0
@nazarhussain
Slow down playwright browser
9aa90ec
@nazarhussain
Update the run scripts for browsers
8b1f4a6
@nazarhussain
Update the run scripts for browsers
8f5d8a1
@nazarhussain
Revert playwright to webdriverio
29772a6
@nazarhussain
Update the timeout
6a3001b
@nazarhussain
Fix types
7066615
@nazarhussain
Fix lint
5d2d006
@nazarhussain
Update package version
11c3546
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@nazarhussain @wemeetagain