Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spec test v1.1.0 #3264

Merged
merged 9 commits into from
Sep 29, 2021
Merged

Spec test v1.1.0 #3264

merged 9 commits into from
Sep 29, 2021

Conversation

twoeths
Copy link
Contributor

@twoeths twoeths commented Sep 28, 2021 •
edited
Loading

Motivation

  • Go with spec test v1.1.0

Description

  • Fix spec tests for phase0, altair and merge
  • Add missing tests
  • Reenable spec test workflow for merge

Close #3252

@codeclimate
Copy link

codeclimate bot commented Sep 28, 2021 •
edited
Loading

Code Climate has analyzed commit e09b6ec and detected 0 issues on this pull request.

View more on Code Climate.

@codecov
Copy link

codecov bot commented Sep 28, 2021 •
edited
Loading

Codecov Report

Merging #3264 (e09b6ec) into master (1fdb7c1) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3264   +/-   ##
=======================================
  Coverage   38.53%   38.53%           
=======================================
  Files         303      303           
  Lines        7624     7624           
  Branches     1120     1120           
=======================================
  Hits         2938     2938           
  Misses       4553     4553           
  Partials      133      133

@github-actions
Copy link
Contributor

github-actions bot commented Sep 28, 2021 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 25f7b92 Previous: ca48f25 Ratio
BeaconState.hashTreeRoot - No change 769.00 ns/op 707.00 ns/op 1.09
BeaconState.hashTreeRoot - 1 full validator 97.890 us/op 77.944 us/op 1.26
BeaconState.hashTreeRoot - 32 full validator 1.4313 ms/op 1.1074 ms/op 1.29
BeaconState.hashTreeRoot - 512 full validator 19.674 ms/op 14.557 ms/op 1.35
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 104.77 us/op 78.731 us/op 1.33
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.4529 ms/op 1.0685 ms/op 1.36
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 22.577 ms/op 16.688 ms/op 1.35
BeaconState.hashTreeRoot - 1 balances 75.599 us/op 55.824 us/op 1.35
BeaconState.hashTreeRoot - 32 balances 648.18 us/op 491.55 us/op 1.32
BeaconState.hashTreeRoot - 512 balances 5.8922 ms/op 4.7784 ms/op 1.23
BeaconState.hashTreeRoot - 250000 balances 114.89 ms/op 89.840 ms/op 1.28
processSlot - 1 slots 63.894 us/op 42.086 us/op 1.52
processSlot - 32 slots 2.9886 ms/op 2.2046 ms/op 1.36
getCommitteeAssignments - req 1 vs - 250000 vc 5.1607 ms/op 5.1818 ms/op 1.00
getCommitteeAssignments - req 100 vs - 250000 vc 7.1376 ms/op 7.1674 ms/op 1.00
getCommitteeAssignments - req 1000 vs - 250000 vc 8.1965 ms/op 7.6541 ms/op 1.07
computeProposers - vc 250000 25.913 ms/op 20.757 ms/op 1.25
computeEpochShuffling - vc 250000 207.08 ms/op 186.19 ms/op 1.11
getNextSyncCommittee - vc 250000 425.31 ms/op 341.07 ms/op 1.25
altair processAttestation - 250000 vs - 7PWei normalcase 43.539 ms/op 35.063 ms/op 1.24
altair processAttestation - 250000 vs - 7PWei worstcase 50.666 ms/op 41.638 ms/op 1.22
altair processAttestation - setStatus - 1/6 committees join 12.377 ms/op 9.1773 ms/op 1.35
altair processAttestation - setStatus - 1/3 committees join 25.905 ms/op 19.585 ms/op 1.32
altair processAttestation - setStatus - 1/2 committees join 39.888 ms/op 30.579 ms/op 1.30
altair processAttestation - setStatus - 2/3 committees join 55.635 ms/op 41.120 ms/op 1.35
altair processAttestation - setStatus - 4/5 committees join 66.509 ms/op 49.303 ms/op 1.35
altair processAttestation - setStatus - 100% committees join 82.729 ms/op 63.950 ms/op 1.29
altair processAttestation - updateEpochParticipants - 1/6 committees join 13.118 ms/op 9.7452 ms/op 1.35
altair processAttestation - updateEpochParticipants - 1/3 committees join 27.527 ms/op 20.745 ms/op 1.33
altair processAttestation - updateEpochParticipants - 1/2 committees join 25.160 ms/op 21.100 ms/op 1.19
altair processAttestation - updateEpochParticipants - 2/3 committees join 26.696 ms/op 22.728 ms/op 1.17
altair processAttestation - updateEpochParticipants - 4/5 committees join 31.102 ms/op 24.211 ms/op 1.28
altair processAttestation - updateEpochParticipants - 100% committees join 28.705 ms/op 25.461 ms/op 1.13
altair processAttestation - updateAllStatus 20.554 ms/op 18.884 ms/op 1.09
altair processBlock - 250000 vs - 7PWei normalcase 50.125 ms/op 42.376 ms/op 1.18
altair processBlock - 250000 vs - 7PWei worstcase 142.88 ms/op 112.20 ms/op 1.27
altair processEpoch - pyrmont_e62330 488.27 ms/op 448.69 ms/op 1.09
pyrmont_e62330 - altair beforeProcessEpoch 168.05 ms/op 152.92 ms/op 1.10
pyrmont_e62330 - altair processJustificationAndFinalization 117.97 us/op 106.59 us/op 1.11
pyrmont_e62330 - altair processInactivityUpdates 8.0418 ms/op 7.9675 ms/op 1.01
pyrmont_e62330 - altair processRewardsAndPenalties 57.064 ms/op 56.274 ms/op 1.01
pyrmont_e62330 - altair processRegistryUpdates 22.876 us/op 15.558 us/op 1.47
pyrmont_e62330 - altair processSlashings 6.2530 us/op 2.5570 us/op 2.45
pyrmont_e62330 - altair processEth1DataReset 6.0870 us/op 2.3130 us/op 2.63
pyrmont_e62330 - altair processEffectiveBalanceUpdates 5.7644 ms/op 6.5020 ms/op 0.89
pyrmont_e62330 - altair processSlashingsReset 32.012 us/op 17.977 us/op 1.78
pyrmont_e62330 - altair processRandaoMixesReset 43.610 us/op 25.537 us/op 1.71
pyrmont_e62330 - altair processHistoricalRootsUpdate 7.8380 us/op 2.8990 us/op 2.70
pyrmont_e62330 - altair processParticipationFlagUpdates 47.240 ms/op 44.035 ms/op 1.07
pyrmont_e62330 - altair processSyncCommitteeUpdates 5.4820 us/op 2.2860 us/op 2.40
pyrmont_e62330 - altair afterProcessEpoch 125.25 ms/op 115.20 ms/op 1.09
altair processInactivityUpdates - 250000 normalcase 77.897 ms/op 65.013 ms/op 1.20
altair processInactivityUpdates - 250000 worstcase 71.899 ms/op 64.676 ms/op 1.11
altair processParticipationFlagUpdates - 250000 anycase 89.689 ms/op 90.381 ms/op 0.99
altair processRewardsAndPenalties - 250000 normalcase 119.18 ms/op 115.46 ms/op 1.03
altair processRewardsAndPenalties - 250000 worstcase 132.37 ms/op 126.38 ms/op 1.05
altair processSyncCommitteeUpdates - 250000 443.67 ms/op 358.94 ms/op 1.24
Tree 40 250000 create 619.23 ms/op 456.79 ms/op 1.36
Tree 40 250000 get(125000) 285.16 ns/op 266.42 ns/op 1.07
Tree 40 250000 set(125000) 2.0688 us/op 1.5028 us/op 1.38
Tree 40 250000 toArray() 44.984 ms/op 37.532 ms/op 1.20
Tree 40 250000 iterate all - toArray() + loop 45.697 ms/op 36.932 ms/op 1.24
Tree 40 250000 iterate all - get(i) 117.23 ms/op 103.39 ms/op 1.13
MutableVector 250000 create 25.519 ms/op 21.238 ms/op 1.20
MutableVector 250000 get(125000) 14.793 ns/op 13.072 ns/op 1.13
MutableVector 250000 set(125000) 752.34 ns/op 566.71 ns/op 1.33
MutableVector 250000 toArray() 9.4083 ms/op 7.4556 ms/op 1.26
MutableVector 250000 iterate all - toArray() + loop 9.7723 ms/op 7.8849 ms/op 1.24
MutableVector 250000 iterate all - get(i) 3.8431 ms/op 3.2760 ms/op 1.17
Array 250000 create 6.5773 ms/op 5.3064 ms/op 1.24
Array 250000 clone - spread 3.0729 ms/op 1.5695 ms/op 1.96
Array 250000 get(125000) 0.91500 ns/op 1.1200 ns/op 0.82
Array 250000 set(125000) 0.99400 ns/op 0.79500 ns/op 1.25
Array 250000 iterate all - loop 141.59 us/op 168.46 us/op 0.84
aggregationBits - 2048 els - readonlyValues 300.93 us/op 208.87 us/op 1.44
aggregationBits - 2048 els - zipIndexesInBitList 65.839 us/op 37.040 us/op 1.78
ssz.Root.equals 1.7280 us/op 1.2170 us/op 1.42
ssz.Root.equals with valueOf() 1.9760 us/op 1.3870 us/op 1.42
byteArrayEquals with valueOf() 2.0230 us/op 1.3940 us/op 1.45
phase0 processBlock - 250000 vs - 7PWei normalcase 18.586 ms/op 11.608 ms/op 1.60
phase0 processBlock - 250000 vs - 7PWei worstcase 105.56 ms/op 74.223 ms/op 1.42
phase0 afterProcessEpoch - 250000 vs - 7PWei 218.31 ms/op 208.55 ms/op 1.05
phase0 beforeProcessEpoch - 250000 vs - 7PWei 814.90 ms/op 576.91 ms/op 1.41
phase0 processEpoch - mainnet_e58758 1.0153 s/op 797.23 ms/op 1.27
mainnet_e58758 - phase0 beforeProcessEpoch 629.82 ms/op 481.83 ms/op 1.31
mainnet_e58758 - phase0 processJustificationAndFinalization 137.62 us/op 122.77 us/op 1.12
mainnet_e58758 - phase0 processRewardsAndPenalties 88.632 ms/op 80.567 ms/op 1.10
mainnet_e58758 - phase0 processRegistryUpdates 90.319 us/op 87.309 us/op 1.03
mainnet_e58758 - phase0 processSlashings 6.5570 us/op 2.4000 us/op 2.73
mainnet_e58758 - phase0 processEth1DataReset 6.3700 us/op 2.7130 us/op 2.35
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 10.595 ms/op 9.7981 ms/op 1.08
mainnet_e58758 - phase0 processSlashingsReset 31.393 us/op 18.940 us/op 1.66
mainnet_e58758 - phase0 processRandaoMixesReset 43.386 us/op 26.095 us/op 1.66
mainnet_e58758 - phase0 processHistoricalRootsUpdate 8.6580 us/op 3.0570 us/op 2.83
mainnet_e58758 - phase0 processParticipationRecordUpdates 27.860 us/op 20.413 us/op 1.36
mainnet_e58758 - phase0 afterProcessEpoch 195.34 ms/op 187.84 ms/op 1.04
phase0 processEffectiveBalanceUpdates - 250000 normalcase 12.089 ms/op 11.662 ms/op 1.04
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.7691 s/op 1.2295 s/op 1.44
phase0 processRegistryUpdates - 250000 normalcase 95.396 us/op 76.981 us/op 1.24
phase0 processRegistryUpdates - 250000 badcase_full_deposits 4.4358 ms/op 3.6612 ms/op 1.21
phase0 processRegistryUpdates - 250000 worstcase 0.5 2.2829 s/op 1.5206 s/op 1.50
phase0 getAttestationDeltas - 250000 normalcase 37.225 ms/op 34.993 ms/op 1.06
phase0 getAttestationDeltas - 250000 worstcase 37.144 ms/op 35.577 ms/op 1.04
phase0 processSlashings - 250000 worstcase 40.866 ms/op 30.875 ms/op 1.32
shuffle list - 16384 els 13.741 ms/op 12.999 ms/op 1.06
shuffle list - 250000 els 190.48 ms/op 188.82 ms/op 1.01
getEffectiveBalances - 250000 vs - 7PWei 13.957 ms/op 10.011 ms/op 1.39
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.7853 ms/op 2.0679 ms/op 1.35
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 1.1081 ms/op 956.95 us/op 1.16
BLS verify - blst-native 2.4001 ms/op 1.8567 ms/op 1.29
BLS verifyMultipleSignatures 3 - blst-native 5.0685 ms/op 3.8121 ms/op 1.33
BLS verifyMultipleSignatures 8 - blst-native 10.960 ms/op 8.2222 ms/op 1.33
BLS verifyMultipleSignatures 32 - blst-native 40.485 ms/op 29.828 ms/op 1.36
BLS aggregatePubkeys 32 - blst-native 57.928 us/op 40.232 us/op 1.44
BLS aggregatePubkeys 128 - blst-native 212.68 us/op 155.56 us/op 1.37
getAttestationsForBlock 105.49 ms/op 76.808 ms/op 1.37
CheckpointStateCache - add get delete 19.504 us/op 14.876 us/op 1.31
validate gossip signedAggregateAndProof - struct 6.0334 ms/op 4.4441 ms/op 1.36
validate gossip signedAggregateAndProof - treeBacked 6.2502 ms/op 4.4560 ms/op 1.40
validate gossip attestation - struct 2.7958 ms/op 2.0772 ms/op 1.35
validate gossip attestation - treeBacked 2.8615 ms/op 2.0939 ms/op 1.37

by benchmarkbot/action

dapplion
dapplion approved these changes Sep 28, 2021
View reviewed changes
Copy link
Contributor

@dapplion dapplion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! Thank you

dapplion
dapplion suggested changes Sep 28, 2021
View reviewed changes
Copy link
Contributor

@dapplion dapplion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh! Spec test did not run

@dapplion dapplion merged commit ccce501 into master Sep 29, 2021
@dapplion dapplion deleted the tuyen/fix-spec-tests-v1.1.0-beta.5 branch September 29, 2021 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dapplion dapplion dapplion approved these changes

@mpetrunic mpetrunic Awaiting requested review from mpetrunic

@wemeetagain wemeetagain Awaiting requested review from wemeetagain

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add merge spec tests v1.1.0
2 participants
@twoeths @dapplion