Skip to content

Commit

Permalink
Bring reference_hardware.json inline with machine used for weights (#…
Browse files Browse the repository at this point in the history
…5196)

Since `May 2023` after
paritytech/substrate#13548 optimization,
`Blake2256` is faster with about 30%, that means that there is a
difference of ~30% between the benchmark values we ask validators to run
against and the machine we use for generating the weights.So if all
validators, just barely pass the benchmarks our weights are potentially
underestimated with about ~20%, so let's bring this two in sync.

Same thing happened when we merged
#2524 in `Nov 2023`
SR25519-Verify became faster with about 10-15%

## Results

Generated on machine from here:
paritytech/devops#3210
```
+----------+----------------+--------------+-------------+-------------------+
| Category | Function       | Score        | Minimum     | Result            |
+============================================================================+
| CPU      | BLAKE2-256     | 1.00 GiBs    | 783.27 MiBs | ✅ Pass (130.7 %) |
|----------+----------------+--------------+-------------+-------------------|
| CPU      | SR25519-Verify | 637.62 KiBs  | 560.67 KiBs | ✅ Pass (113.7 %) |
|----------+----------------+--------------+-------------+-------------------|
| Memory   | Copy           | 12.19 GiBs   | 11.49 GiBs  | ✅ Pass (106.1 %) |
```

Discovered and discussed here:
#5127 (comment)

## Downsides

Machines that barely passed the benchmark will suddenly find themselves
bellow the benchmark, but since that is just an warning and everything
else continues as before it shouldn't be too impactful and should give
the validators the necessary information that they need to become
compliant, since they actually aren't when compared with the used
weights.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
  • Loading branch information
alexggh authored Aug 7, 2024
1 parent efdc1e9 commit 8af9889
Show file tree
Hide file tree
Showing 3 changed files with 27 additions and 4 deletions.
23 changes: 23 additions & 0 deletions prdoc/pr_5196.prdoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Schema: Polkadot SDK PRDoc Schema (prdoc) v1.0.0
# See doc at https://raw.githubusercontent.com/paritytech/polkadot-sdk/master/prdoc/schema_user.json

title: Bring benchmark inline with reference machine used for weights

doc:
- audience: Node Operator
description: |
- BLAKE2-256 reference values were too low(~30%) when compared with the machine used for generating
the weights, so it was brought in sync with results on the reference hardware recommended here:
https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot#reference-hardware
- SR25519-Verify reference values were too low(~10%) when compared with the machine used for generating
the weights, so it was brought in sync with results on the reference hardware recommended here:
https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot#reference-hardware
- Validators where the `BLAKE2-256` and `SR25519-Verify` were barely passing, might received the
warning that they are not compliant anymore, this should not be treated as critical, but they
should take the necessary steps to become compliant in the near/mid-term future.
- Note!: The reference hardware requirements have not been increased we just fixed the benchmark which
was wrongly reporting lower spec HW as being compliant.

crates:
- name: frame-benchmarking-cli
bump: minor
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,10 @@ mod tests {
assert_eq!(
*SUBSTRATE_REFERENCE_HARDWARE,
Requirements(vec![
Requirement { metric: Metric::Blake2256, minimum: Throughput::from_mibs(783.27) },
Requirement { metric: Metric::Blake2256, minimum: Throughput::from_mibs(1000.00) },
Requirement {
metric: Metric::Sr25519Verify,
minimum: Throughput::from_kibs(560.670000128),
minimum: Throughput::from_kibs(637.619999744),
},
Requirement {
metric: Metric::MemCopy,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
[
{
"metric": "Blake2256",
"minimum": 783.27
"minimum": 1000.00
},
{
"metric": "Sr25519Verify",
"minimum": 0.547529297
"minimum": 0.622675781
},
{
"metric": "MemCopy",
Expand Down

0 comments on commit 8af9889

Please sign in to comment.