Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run benchmark script once #2778

Merged
merged 8 commits into from
Jun 30, 2021
Merged

Run benchmark script once #2778

merged 8 commits into from
Jun 30, 2021

Conversation

dapplion
Copy link
Contributor

Motivation

We must not run benchmark script twice:

  • It won't persist all benchmark data but only the data associated to the last package
  • Therefore there won't be any data to compare for all packages but the last

Description

  • Run benchmark runner once with a glob that grabs perf tests from all packages, without lerna.
  • Simplify BLS initialization to work with this approach
  • Run benchmark Github action not in forks
  • Disable unnecessary benchmarks in CI
  • Run CheckpointStateCache initialization in before block
  • Add basic BLS performance test

@codeclimate
Copy link

codeclimate bot commented Jun 30, 2021

Code Climate has analyzed commit 73e777b and detected 0 issues on this pull request.

View more on Code Climate.

@github-actions
Copy link
Contributor

Performance Report

✔️ no performance regression detected

🚀🚀 Significant benchmark improvement detected

Benchmark suite Current: 89c57b9 Previous: bad3f6e Ratio
Process block - 250000 vs - with 0 validator exit 224.98 us/op 5.7165 ms/op 0.04
epoch - 250000 vs - processSlashings 33.791 us/op 3.5590 ms/op 0.01
shuffle list - 512 els 144.20 us/op 1.4239 ms/op 0.10
shuffle list - 16384 els 1.8196 ms/op 15.937 ms/op 0.11
shuffle list - 250000 els 24.698 ms/op 225.32 ms/op 0.11
Full benchmark results
Benchmark suite Current: 89c57b9 Previous: bad3f6e Ratio
Process block - 250000 vs - with 0 validator exit 224.98 us/op 5.7165 ms/op 0.04
Process block - 250000 vs - with 1 validator exit 20.519 ms/op 25.965 ms/op 0.79
Process block - 250000 vs - with 16 validator exits 21.978 ms/op 26.717 ms/op 0.82
epoch - 250000 vs - processJustificationAndFinalization 69.331 us/op 119.72 us/op 0.58
epoch - 250000 vs - processRewardsAndPenalties 275.41 ms/op 359.30 ms/op 0.77
epoch - 250000 vs - processRegistryUpdates 9.9640 us/op 10.317 us/op 0.97
epoch - 250000 vs - processSlashings 33.791 us/op 3.5590 ms/op 0.01
epoch - 250000 vs - processFinalUpdates 39.957 ms/op 39.089 ms/op 1.02
epoch - 250000 vs - prepareEpochProcessState 543.03 ms/op 1.0793 s/op 0.50
getAttestationDeltas - 250000 vs 110.40 ms/op 165.88 ms/op 0.67
processSlots - 250000 vs - 32 empty slots 3.5484 s/op 1.9698 s/op 1.80
shuffle list - 512 els 144.20 us/op 1.4239 ms/op 0.10
shuffle list - 16384 els 1.8196 ms/op 15.937 ms/op 0.11
shuffle list - 250000 els 24.698 ms/op 225.32 ms/op 0.11
aggregationBits - 2048 els - readonlyValues 235.97 us/op 273.96 us/op 0.86
aggregationBits - 2048 els - zipIndexesInBitList 43.099 us/op 56.178 us/op 0.77
ssz.Root.equals 1.2270 us/op 1.2810 us/op 0.96
BLS verify - blst-native 1.9296 ms/op
BLS verifyMultipleSignatures 3 - blst-native 3.9775 ms/op
BLS verifyMultipleSignatures 8 - blst-native 8.5658 ms/op
BLS verifyMultipleSignatures 32 - blst-native 33.578 ms/op
BLS aggregatePubkeys 32 - blst-native 42.014 us/op
BLS aggregatePubkeys 128 - blst-native 161.04 us/op
CheckpointStateCache - add get delete 34.588 us/op

by benchmarkbot/action

@wemeetagain wemeetagain merged commit 5173df2 into master Jun 30, 2021
@wemeetagain wemeetagain deleted the dapplion/run-one-benchmark branch June 30, 2021 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants