Add storage bounds for pallet `staking` and clean up deprecated non paged exposure storages #6445

re-gius · 2024-11-11T15:08:49Z

This is part of #6289 and necessary for the Asset Hub migration.

Building on the observations and suggestions from #255 .

Changes

Add MaxInvulnerables to bound Invulnerables Vec -> BoundedVec.
- Set to constant 20 in the pallet (must be >= 17 for backward compatibility of runtime westend).
Add MaxDisabledValidators to bound DisabledValidators Vec -> BoundedVec
- Set to constant 100 in the pallet (it should be <= 1/3 * MaxValidatorsCount according to the current disabling strategy)
Remove ErasStakers and ErasStakersClipped (see Tracker issue for cleaning up old non-paged exposure logic in staking pallet #433 )
- They were deprecated in v14 and could have been removed since staking era 1504 (now it's > 1600)
- Completing the task from [pallet-staking] Clean up stale non paged exposure storages in the pallet #5986
Use MaxExposurePageSize to bound ErasStakersPaged mapping to exposure pages: each ExposurePage.others Vec is turned into a WeakBoundedVec to allow easy and quick changes to this bound
Add MaxBondedEras to bound BondedEras Vec -> BoundedVec
- Set to BondingDuration::get() + 1 everywhere to include both time interval endpoints in [current_era - BondingDuration::get(), current_era]. Notice that this was done manually in every test and runtime, so I wonder if there is a better way to ensure that MaxBondedEras::get() == BondingDuration::get() + 1 everywhere.
Add MaxRewardPagesPerValidator to bound ClaimedRewards Vec of pages -> WeakBoundedVec
- Set to constant 20 in the pallet. The vector of pages is now a WeakBoundedVec to allow easy and quick changes to this parameter
Remove MaxValidatorsCount optional storage item to add MaxValidatorsCount mandatory config parameter
- Using it to to bound EraRewardPoints.individual BTreeMap -> BoundedBTreeMap;
- Set to dynamic parameter in runtime westend so that changing it should not require migrations for it

TO DO
Slashing storage items will be bounded in another PR.

UnappliedSlashes
SlashingSpans

gui1117 · 2024-12-19T02:46:29Z

substrate/primitives/staking/src/lib.rs

-		ExposurePage { page_total: Default::default(), others: vec![] }
+		ExposurePage {
+			page_total: Default::default(),
+			others: WeakBoundedVec::force_from(vec![], None),


if you want you can create a fn new method on WeakBoundedVec

Yes, I just opened this PR

paritytech/parity-common#888

Just call default.

Ok, changed to default() in 4b58f0a

gui1117 · 2024-12-19T02:54:21Z

substrate/primitives/staking/src/lib.rs

 {
 	/// Splits an `Exposure` into `PagedExposureMetadata` and multiple chunks of
 	/// `IndividualExposure` with each chunk having maximum of `page_size` elements.
-	pub fn into_pages(
+	pub fn into_pages<MaxExposurePageSize>(


This function is only used one time and page_size is equal to MaxExposurePageSize. I think we should refactor: remove the argument page_size.

If needed we can always introduce a try_into_pages with an argument.
In general I feel it is not so good to add many implicit constraint, like here page_size must be less than MaxExposurePageSize otherwise some element will be ignored (with a log::error, but still).

Changed in 489fc9d

kianenigma · 2024-12-19T14:19:05Z

polkadot/runtime/rococo/src/lib.rs

@@ -1184,7 +1184,6 @@ impl parachains_slashing::Config for Runtime {
 		ReportLongevity,
 	>;
 	type WeightInfo = parachains_slashing::TestWeightInfo;
-	type BenchmarkingConfig = parachains_slashing::BenchConfig<200>;


The overall observation is that we had to add temporarily a lot of bounds to pallets just for benchmarks, but as staking has now proper bounds for all its storage, they are not needed ✅

polkadot/runtime/test-runtime/src/lib.rs

kianenigma · 2024-12-19T14:20:57Z

polkadot/runtime/westend/src/genesis_config_presets.rs

+			invulnerables: BoundedVec::try_from(
+				initial_authorities.iter().map(|x| x.0.clone()).collect::<Vec<_>>()
+			)
+			.expect("Too many invulnerable validators!"),


Error message can be a bit more informative, hinting at which config should be tweaked if this expect is ever reached.

error message improved in 0d47dc5

kianenigma · 2024-12-19T14:24:20Z

polkadot/runtime/westend/src/lib.rs

@@ -1841,6 +1855,7 @@ pub mod migrations {
 		parachains_shared::migration::MigrateToV1<Runtime>,
 		parachains_scheduler::migration::MigrateV2ToV3<Runtime>,
 		pallet_staking::migrations::v16::MigrateV15ToV16<Runtime>,
+		pallet_staking::migrations::v17::MigrateV16ToV17<Runtime>,


You can check the logs for this migration in this CI job:

https://github.com/paritytech/polkadot-sdk/actions/runs/12412281493/job/34651846644?pr=6445

there seems to be some error-ish logs in there:

💸 Migration failed for ClaimedRewards from v16 to v17.

Solved with force_from(...) for WeakBoundedVec. I noticed that old individual validator exposures have up to 205 pages, while we would expect them to be at most 20 in the future @Ank4n
What do you think is the best approach: keep 20 as a limit for MaxRewardPagesPerValidator for future items and force old items in a WeakBoundedVec or increase the MaxRewardPagesPerValidator limit to something like 250 to accomodate old items?

EDIT: I can also see a third option: merge pages for old items until they are less than 20.

kianenigma · 2024-12-19T14:30:22Z

substrate/bin/node/runtime/src/lib.rs

@@ -745,6 +746,10 @@ impl pallet_staking::Config for Runtime {
 	type WeightInfo = pallet_staking::weights::SubstrateWeight<Runtime>;
 	type BenchmarkingConfig = StakingBenchmarkingConfig;
 	type DisablingStrategy = pallet_staking::UpToLimitWithReEnablingDisablingStrategy;
+	type MaxInvulnerables = ConstU32<20>;
+	type MaxRewardPagesPerValidator = ConstU32<20>;
+	type MaxValidatorsCount = ConstU32<300>;


Other pallets like session, babe, grandpa and such also have a notion of MaxValidators (expressed as MaxAuthorities) that should be equal to the max validators of this pallet.

within this file, you can use one pub const MaxValidators: u32 = 300 everywhere to unify it.

Taking it a step further, you can expose this as a part of trait SessionManager. this trait can expose a trat SessionManager { type MaxAuthorities = <set-by-staking-pallet> }. Then, within the pallet-session, who consumes SessionManager, you can do:

fn integroty_check() { /// A way to express within pallet-session that whoever implements `SessionManager` should have a compatible `MaxAuthorities`. assert!(T::SessionManager::MaxAuthoritiet::get() >= T::MaxAuthorities::get()) }

This might be too much for this PR, but good for you to be familiar with the pattern.

Whenever they are multiple parameters within two pallets that have a logical dependency (they have to be equal, or one has to be larger than the other), you can remove the implicitness like this.

changed in a065a91

Opened issue #6972 to unify MaxValidators limit (I also see a MaxActiveValidators)

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

paritytech-workflow-stopper · 2024-12-20T12:35:36Z

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/12431335243
Failed job name: cargo-clippy

add MaxInvulnerables

6c808b6

re-gius added the T2-pallets This PR/Issue is related to a particular pallet. label Nov 11, 2024

re-gius self-assigned this Nov 11, 2024

This was referenced Nov 11, 2024

[pallet-staking] Bound all storage #5987

Closed

Tracking issue for bounding pallet-staking storage items #255

Open

[tracking] remove without_storage_info for pallets migrating to AH #6289

Open

re-gius added 23 commits November 13, 2024 14:51

add MaxActiveValidators

1b18bb1

Merge branch 'master' into re-gius/staking/add-storage-bounds

0ac60af

fix + start working on migrations

4edb8e1

nit (typo)

65c7de6

fix migrations for BoundedVec

35069c9

Merge branch 'master' into re-gius/staking/add-storage-bounds

d4b15af

Migrate ErasRewardPoints

eb93a70

small fixes - BoundedBTreeMap

58cf45d

Merge branch 'master' into re-gius/staking/add-storage-bounds

292c5d0

fix

74f2055

Merge branch 'master' into re-gius/staking/add-storage-bounds

63f2c5e

Merge branch 'master' into re-gius/staking/add-storage-bounds

b249a67

Adapt to new recent DisabledValidators change

362f505

Change MaxActiveValidators to MaxWinners

4f21d53

undo change from another PR

e258baf

Merge branch 'master' into re-gius/staking/add-storage-bounds

4b7d896

small fix for MaxWinners

01e2233

small fixes

243bec8

Merge branch 'master' into re-gius/staking/add-storage-bounds

a599afc

Remove deprecated ErasStakers and ErasStakersClipped

9a25fbc

Fixes from #5986

98daabc

nit

aec5087

remove unused import

36204da

paritytech deleted a comment from paritytech-workflow-stopper bot Nov 25, 2024

fix migrations

831f1b0

gui1117 reviewed Dec 19, 2024

View reviewed changes

prefix with _ unused variables

3702ce5

paritytech-review-bot bot requested a review from a team December 19, 2024 09:31

re-gius added 6 commits December 19, 2024 11:10

remove unused page_size fn arg + fix tests

489fc9d

Merge branch 'master' into re-gius/staking/add-storage-bounds

65fc95b

Merge branch 'master' into re-gius/staking/add-storage-bounds

9c88d39

fix merge

a5e98e6

fix merge

7b82520

fix merge

30520af

kianenigma reviewed Dec 19, 2024

View reviewed changes

polkadot/runtime/test-runtime/src/lib.rs Outdated Show resolved Hide resolved

kianenigma reviewed Dec 19, 2024

View reviewed changes

re-gius added 8 commits December 19, 2024 15:57

expand errors msg for invulnerables

0d47dc5

using default() to create empty WeakBoundedVec

4b58f0a

migration fix attempt

33e59b2

nit

c889905

migration fix

300b7f9

fix migrations

fc63dc7

Merge branch 'master' into re-gius/staking/add-storage-bounds

720c9eb

Set MaxValidatorsCount equal to MaxAuthorities

a065a91

re-gius mentioned this pull request Dec 20, 2024

Unify bounds for the maximum number of validators across pallets #6972

Open

re-gius and others added 4 commits December 20, 2024 13:19

Update polkadot/runtime/test-runtime/src/lib.rs

483eb03

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

Merge branch 'master' into re-gius/staking/add-storage-bounds

7aeb8a5

fix comment

c51d09b

fix

1a9be6b

Update pr_6445.prdoc from semver CI

235a260

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add storage bounds for pallet `staking` and clean up deprecated non paged exposure storages #6445

Add storage bounds for pallet `staking` and clean up deprecated non paged exposure storages #6445

re-gius commented Nov 11, 2024 •

edited

Loading

gui1117 Dec 19, 2024

re-gius Dec 19, 2024

bkchr Dec 19, 2024

re-gius Dec 19, 2024

gui1117 Dec 19, 2024

re-gius Dec 19, 2024

kianenigma Dec 19, 2024

kianenigma Dec 19, 2024

re-gius Dec 19, 2024

kianenigma Dec 19, 2024

re-gius Dec 20, 2024 •

edited

Loading

kianenigma Dec 19, 2024 •

edited

Loading

re-gius Dec 20, 2024

paritytech-workflow-stopper bot commented Dec 20, 2024

Add storage bounds for pallet staking and clean up deprecated non paged exposure storages #6445

Are you sure you want to change the base?

Add storage bounds for pallet staking and clean up deprecated non paged exposure storages #6445

Conversation

re-gius commented Nov 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

re-gius Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

kianenigma Dec 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paritytech-workflow-stopper bot commented Dec 20, 2024

Add storage bounds for pallet `staking` and clean up deprecated non paged exposure storages #6445

Add storage bounds for pallet `staking` and clean up deprecated non paged exposure storages #6445

re-gius commented Nov 11, 2024 •

edited

Loading

re-gius Dec 20, 2024 •

edited

Loading

kianenigma Dec 19, 2024 •

edited

Loading