Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add storage bounds for pallet staking and clean up deprecated non paged exposure storages #6445

Open
wants to merge 135 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
135 commits
Select commit Hold shift + click to select a range
6c808b6
add MaxInvulnerables
re-gius Nov 11, 2024
1b18bb1
add `MaxActiveValidators`
re-gius Nov 13, 2024
0ac60af
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Nov 13, 2024
4edb8e1
fix + start working on migrations
re-gius Nov 13, 2024
65c7de6
nit (typo)
re-gius Nov 13, 2024
35069c9
fix migrations for `BoundedVec`
re-gius Nov 14, 2024
d4b15af
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Nov 14, 2024
eb93a70
Migrate `ErasRewardPoints`
re-gius Nov 14, 2024
58cf45d
small fixes - `BoundedBTreeMap`
re-gius Nov 15, 2024
292c5d0
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Nov 15, 2024
74f2055
fix
re-gius Nov 15, 2024
63f2c5e
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Nov 18, 2024
b249a67
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Nov 22, 2024
362f505
Adapt to new recent `DisabledValidators` change
re-gius Nov 22, 2024
4f21d53
Change `MaxActiveValidators` to `MaxWinners`
re-gius Nov 25, 2024
e258baf
undo change from another PR
re-gius Nov 25, 2024
4b7d896
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Nov 25, 2024
01e2233
small fix for `MaxWinners`
re-gius Nov 25, 2024
243bec8
small fixes
re-gius Nov 25, 2024
a599afc
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Nov 25, 2024
9a25fbc
Remove deprecated `ErasStakers` and `ErasStakersClipped`
re-gius Nov 25, 2024
98daabc
Fixes from #5986
re-gius Nov 25, 2024
aec5087
nit
re-gius Nov 25, 2024
36204da
remove unused import
re-gius Nov 25, 2024
18e9bee
fix `Invulnerables` migration
re-gius Nov 25, 2024
cbfd4a5
Undo migrations change
re-gius Nov 26, 2024
b5cbfec
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Nov 26, 2024
9ca25a1
debug
re-gius Nov 26, 2024
10e7295
Increase `MaxInvulnerables` to 20 + fix migrations msg
re-gius Nov 26, 2024
b3186b1
fix - `MaxInvulnerables` = 20
re-gius Nov 26, 2024
336137a
Fix tests
re-gius Nov 26, 2024
c1911e1
Fix impl
re-gius Nov 26, 2024
2ac794e
Fix `Invulnerables` limit in benchmarking
re-gius Nov 26, 2024
583e46d
Bound `ExposurePage.others` with `MaxExposurePageSize`
re-gius Nov 26, 2024
6347a8c
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Nov 26, 2024
9e82b85
fix `TypeInfo` error
re-gius Nov 27, 2024
1db7f39
fix benchmarking limit
re-gius Nov 27, 2024
f996a97
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Nov 27, 2024
e030506
bound `BondedEras` Vec
re-gius Nov 27, 2024
3ab45d9
Add migration for `BondedEras`
re-gius Nov 27, 2024
a980daa
fix tests attempt
re-gius Nov 27, 2024
827e898
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Nov 28, 2024
c4d5142
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Nov 29, 2024
df8e567
bound `ClaimedRewards`
re-gius Nov 29, 2024
c066632
Use `WeakBoundedVec` for `EraStakersPaged` paged exposures
re-gius Nov 29, 2024
686b6b3
Turn more `BoundedVec` into `WeakBoundedVec`
re-gius Nov 29, 2024
9cf2263
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Nov 29, 2024
21ad9ea
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 2, 2024
ca881ac
add import
re-gius Dec 2, 2024
e986264
Migrate `MaxValidatorsCount` from storage to dynamic param + fixes
re-gius Dec 2, 2024
ea59383
Use `MaxValidatorsCount` to bound `ErasRewardPoints` `BoundedVec`
re-gius Dec 2, 2024
9a21b8f
remove unused imports
re-gius Dec 3, 2024
47863bb
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 3, 2024
50ce1a1
Use `MaxDisabledValidators` to bound disabled validators
re-gius Dec 3, 2024
40ccdec
fix `Vec`/`BoundedVec`
re-gius Dec 3, 2024
c3365a6
Fix `WeakBoundedVec` and `MaxValidatorsCount`
re-gius Dec 3, 2024
584fe53
fix
re-gius Dec 3, 2024
5333a7d
fix
re-gius Dec 3, 2024
31a9300
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 3, 2024
898d469
`BoundedVec` fix in tests
re-gius Dec 3, 2024
32df05d
fix
re-gius Dec 4, 2024
7bc9b98
Use `MaxValidatorsCount` for limit checks
re-gius Dec 4, 2024
9c4d874
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 4, 2024
536d5c5
undo wrong changes
re-gius Dec 9, 2024
608ec88
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 9, 2024
76469ea
Increase benchmarking `MaxValidators` limit
re-gius Dec 9, 2024
fb1a4fa
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 9, 2024
181fe4d
Remove `MaxValidators` from benchmarking config and use `MaxValidator…
re-gius Dec 9, 2024
b1f5f97
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 9, 2024
9c002c1
fix
re-gius Dec 9, 2024
503a7d9
Use `MaxValidatorsCount` for slashing pallet
re-gius Dec 9, 2024
63a18b1
Fix migration comments
re-gius Dec 9, 2024
49059c0
Restore benchmarking for slashing
re-gius Dec 10, 2024
efde7a7
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 10, 2024
8186680
fix
re-gius Dec 10, 2024
3fb6ca7
fix
re-gius Dec 10, 2024
66923a4
fix benchmarking config
re-gius Dec 10, 2024
bec0544
fix
re-gius Dec 10, 2024
3708085
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 10, 2024
73af22a
Remove `unbounded` for `ClaimedRewards`
re-gius Dec 10, 2024
1139689
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 11, 2024
2bda8eb
Update from re-gius running command 'prdoc --bump major --audience ru…
Dec 11, 2024
fe3147b
fix prdoc
re-gius Dec 11, 2024
cd8a0c3
fix prdoc
re-gius Dec 11, 2024
5b15e24
debug failing benchmark
re-gius Dec 11, 2024
9d055a8
benchmarking error debug
re-gius Dec 11, 2024
07b021b
undo debug changes
re-gius Dec 11, 2024
87e0e46
debug 1
re-gius Dec 11, 2024
473e0ff
debug 1
re-gius Dec 11, 2024
eb64ffc
debug 1
re-gius Dec 11, 2024
74d6c7a
debug dynamic param
re-gius Dec 11, 2024
94f5b31
restore after debugging
re-gius Dec 11, 2024
7b1fc5e
Fix bound comparison to `MaxValidatorsCount`
re-gius Dec 12, 2024
cf9d8c3
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 12, 2024
c265974
Use `MaxValidatorsCount` limit to check validators
re-gius Dec 16, 2024
bb5229c
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 16, 2024
6588714
fix staking tests
re-gius Dec 16, 2024
4754629
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 16, 2024
76a2d4b
Fix staking tests
re-gius Dec 17, 2024
67c41fa
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 17, 2024
bd28db5
fix benchmark
re-gius Dec 17, 2024
043b7d6
comment nit
re-gius Dec 17, 2024
68481f4
Always link `BondingDuration` to `MaxBondedEras` in tests
re-gius Dec 17, 2024
312904e
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 17, 2024
d508f8d
Remove old `MaxValidatorCount` from expected default genesis config
re-gius Dec 18, 2024
529fa0f
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 18, 2024
17eeef7
Update cumulus/parachains/integration-tests/emulated/chains/relays/we…
re-gius Dec 18, 2024
21a12d1
Fix PR title in prdoc and migrations for old deprecated storage items
re-gius Dec 18, 2024
e2fa0ca
Update substrate/frame/staking/src/lib.rs
re-gius Dec 18, 2024
4bf3cd6
add debug msg
re-gius Dec 18, 2024
f7c1fe4
fix migrations
re-gius Dec 18, 2024
c881bb0
fix comments + CR
re-gius Dec 18, 2024
306cd41
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 18, 2024
2da736a
Add more debugging `expect(...)` in tests
re-gius Dec 18, 2024
831f1b0
fix migrations
re-gius Dec 18, 2024
3702ce5
prefix with `_` unused variables
re-gius Dec 19, 2024
489fc9d
remove unused `page_size` fn arg + fix tests
re-gius Dec 19, 2024
65fc95b
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 19, 2024
9c88d39
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 19, 2024
a5e98e6
fix merge
re-gius Dec 19, 2024
7b82520
fix merge
re-gius Dec 19, 2024
30520af
fix merge
re-gius Dec 19, 2024
0d47dc5
expand errors msg for invulnerables
re-gius Dec 19, 2024
4b58f0a
using `default()` to create empty `WeakBoundedVec`
re-gius Dec 19, 2024
33e59b2
migration fix attempt
re-gius Dec 19, 2024
c889905
nit
re-gius Dec 19, 2024
300b7f9
migration fix
re-gius Dec 19, 2024
fc63dc7
fix migrations
re-gius Dec 20, 2024
720c9eb
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 20, 2024
a065a91
Set `MaxValidatorsCount` equal to `MaxAuthorities`
re-gius Dec 20, 2024
483eb03
Update polkadot/runtime/test-runtime/src/lib.rs
re-gius Dec 20, 2024
7aeb8a5
Merge branch 'master' into re-gius/staking/add-storage-bounds
re-gius Dec 20, 2024
c51d09b
fix comment
re-gius Dec 20, 2024
1a9be6b
fix
re-gius Dec 20, 2024
235a260
Update pr_6445.prdoc from semver CI
re-gius Dec 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion polkadot/runtime/common/src/try_runtime.rs
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ where

let all_stakers = Ledger::<T>::iter().map(|(ctrl, l)| (ctrl, l.stash)).collect::<BTreeSet<_>>();
let mut all_exposed = BTreeSet::new();
ErasStakers::<T>::iter().for_each(|(_, val, expo)| {
ErasStakersPaged::<T>::iter().for_each(|((_era, val, _page), expo)| {
all_exposed.insert(val);
all_exposed.extend(expo.others.iter().map(|ie| ie.who.clone()))
});
Expand Down
3 changes: 3 additions & 0 deletions polkadot/runtime/test-runtime/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -340,6 +340,7 @@ parameter_types! {
pub storage SessionsPerEra: SessionIndex = 6;
// 28 eras for unbonding (7 days).
pub storage BondingDuration: sp_staking::EraIndex = 28;
pub storage MaxBondedEras: u32 = (BondingDuration::get() as u32) + 1;
re-gius marked this conversation as resolved.
Show resolved Hide resolved
// 27 eras in which slashes can be cancelled (a bit less than 7 days).
pub storage SlashDeferDuration: sp_staking::EraIndex = 27;
pub const RewardCurve: &'static PiecewiseLinear<'static> = &REWARD_CURVE;
Expand Down Expand Up @@ -376,6 +377,7 @@ impl pallet_staking::Config for Runtime {
type Reward = ();
type SessionsPerEra = SessionsPerEra;
type BondingDuration = BondingDuration;
type MaxBondedEras = MaxBondedEras;
type SlashDeferDuration = SlashDeferDuration;
type AdminOrigin = frame_system::EnsureNever<()>;
type SessionInterface = Self;
Expand All @@ -396,6 +398,7 @@ impl pallet_staking::Config for Runtime {
type EventListeners = ();
type WeightInfo = ();
type DisablingStrategy = pallet_staking::UpToLimitWithReEnablingDisablingStrategy;
type MaxInvulnerables = ConstU32<20>;
}

parameter_types! {
Expand Down
4 changes: 4 additions & 0 deletions polkadot/runtime/westend/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -716,6 +716,7 @@ parameter_types! {
pub const SessionsPerEra: SessionIndex = prod_or_fast!(6, 1);
// 2 eras for unbonding (12 hours).
pub const BondingDuration: sp_staking::EraIndex = 2;
pub const MaxBondedEras: u32 = (BondingDuration::get() as u32) + 1;
// 1 era in which slashes can be cancelled (6 hours).
pub const SlashDeferDuration: sp_staking::EraIndex = 1;
pub const MaxExposurePageSize: u32 = 64;
Expand All @@ -738,6 +739,7 @@ impl pallet_staking::Config for Runtime {
type Reward = ();
type SessionsPerEra = SessionsPerEra;
type BondingDuration = BondingDuration;
type MaxBondedEras = MaxBondedEras;
type SlashDeferDuration = SlashDeferDuration;
type AdminOrigin = EitherOf<EnsureRoot<AccountId>, StakingAdmin>;
type SessionInterface = Self;
Expand All @@ -756,6 +758,7 @@ impl pallet_staking::Config for Runtime {
type EventListeners = (NominationPools, DelegatedStaking);
type WeightInfo = weights::pallet_staking::WeightInfo<Runtime>;
type DisablingStrategy = pallet_staking::UpToLimitWithReEnablingDisablingStrategy;
type MaxInvulnerables = frame_support::traits::ConstU32<20>;
}

impl pallet_fast_unstake::Config for Runtime {
Expand Down Expand Up @@ -1839,6 +1842,7 @@ pub mod migrations {
parachains_shared::migration::MigrateToV1<Runtime>,
parachains_scheduler::migration::MigrateV2ToV3<Runtime>,
pallet_staking::migrations::v16::MigrateV15ToV16<Runtime>,
pallet_staking::migrations::v17::MigrateV16ToV17<Runtime>,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can check the logs for this migration in this CI job:

https://github.com/paritytech/polkadot-sdk/actions/runs/12412281493/job/34651846644?pr=6445

there seems to be some error-ish logs in there:

💸 Migration failed for ClaimedRewards from v16 to v17.

Copy link
Contributor Author

@re-gius re-gius Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solved with force_from(...) for WeakBoundedVec. I noticed that old individual validator exposures have up to 205 pages, while we would expect them to be at most 20 in the future @Ank4n
What do you think is the best approach: keep 20 as a limit for MaxRewardPagesPerValidator for future items and force old items in a WeakBoundedVec or increase the MaxRewardPagesPerValidator limit to something like 250 to accomodate old items?

EDIT: I can also see a third option: merge pages for old items until they are less than 20.

// permanent
pallet_xcm::migration::MigrateToLatestXcmVersion<Runtime>,
);
Expand Down
2 changes: 0 additions & 2 deletions polkadot/runtime/westend/src/weights/pallet_fast_unstake.rs
Original file line number Diff line number Diff line change
Expand Up @@ -108,8 +108,6 @@ impl<T: frame_system::Config> pallet_fast_unstake::WeightInfo for WeightInfo<T>
/// Proof Skipped: ElectionProviderMultiPhase CurrentPhase (max_values: Some(1), max_size: None, mode: Measured)
/// Storage: Staking CurrentEra (r:1 w:0)
/// Proof: Staking CurrentEra (max_values: Some(1), max_size: Some(4), added: 499, mode: MaxEncodedLen)
/// Storage: Staking ErasStakers (r:257 w:0)
/// Proof Skipped: Staking ErasStakers (max_values: None, max_size: None, mode: Measured)
/// The range of component `v` is `[1, 256]`.
/// The range of component `b` is `[1, 64]`.
fn on_idle_check(v: u32, b: u32, ) -> Weight {
Expand Down
2 changes: 0 additions & 2 deletions polkadot/runtime/westend/src/weights/pallet_staking.rs
Original file line number Diff line number Diff line change
Expand Up @@ -488,8 +488,6 @@ impl<T: frame_system::Config> pallet_staking::WeightInfo for WeightInfo<T> {
/// Proof: `Staking::Bonded` (`max_values`: None, `max_size`: Some(72), added: 2547, mode: `MaxEncodedLen`)
/// Storage: `Staking::Ledger` (r:65 w:65)
/// Proof: `Staking::Ledger` (`max_values`: None, `max_size`: Some(1091), added: 3566, mode: `MaxEncodedLen`)
/// Storage: `Staking::ErasStakersClipped` (r:1 w:0)
/// Proof: `Staking::ErasStakersClipped` (`max_values`: None, `max_size`: None, mode: `Measured`)
/// Storage: `Staking::ErasStakersOverview` (r:1 w:0)
/// Proof: `Staking::ErasStakersOverview` (`max_values`: None, `max_size`: Some(92), added: 2567, mode: `MaxEncodedLen`)
/// Storage: `Staking::ClaimedRewards` (r:1 w:1)
Expand Down
3 changes: 3 additions & 0 deletions substrate/bin/node/runtime/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -694,6 +694,7 @@ pallet_staking_reward_curve::build! {
parameter_types! {
pub const SessionsPerEra: sp_staking::SessionIndex = 6;
pub const BondingDuration: sp_staking::EraIndex = 24 * 28;
pub const MaxBondedEras: u32 = (BondingDuration::get() as u32) + 1;
pub const SlashDeferDuration: sp_staking::EraIndex = 24 * 7; // 1/4 the bonding duration.
pub const RewardCurve: &'static PiecewiseLinear<'static> = &REWARD_CURVE;
pub const MaxNominators: u32 = 64;
Expand Down Expand Up @@ -722,6 +723,7 @@ impl pallet_staking::Config for Runtime {
type Reward = (); // rewards are minted from the void
type SessionsPerEra = SessionsPerEra;
type BondingDuration = BondingDuration;
type MaxBondedEras = MaxBondedEras;
type SlashDeferDuration = SlashDeferDuration;
/// A super-majority of the council can cancel the slash.
type AdminOrigin = EitherOfDiverse<
Expand All @@ -745,6 +747,7 @@ impl pallet_staking::Config for Runtime {
type WeightInfo = pallet_staking::weights::SubstrateWeight<Runtime>;
type BenchmarkingConfig = StakingBenchmarkingConfig;
type DisablingStrategy = pallet_staking::UpToLimitWithReEnablingDisablingStrategy;
type MaxInvulnerables = ConstU32<20>;
}

impl pallet_fast_unstake::Config for Runtime {
Expand Down
4 changes: 0 additions & 4 deletions substrate/frame/fast-unstake/src/weights.rs

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

31 changes: 24 additions & 7 deletions substrate/frame/staking/src/benchmarking.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,11 @@ use crate::{asset, ConfigOp, Pallet as Staking};
use testing_utils::*;

use codec::Decode;
use frame_election_provider_support::{bounds::DataProviderBounds, SortedListProvider};
use frame_election_provider_support::{
bounds::DataProviderBounds, ElectionProviderBase, SortedListProvider,
};
use frame_support::{
assert_ok,
pallet_prelude::*,
storage::bounded_vec::BoundedVec,
traits::{Get, Imbalance, UnfilteredDispatchable},
Expand Down Expand Up @@ -122,10 +125,17 @@ pub fn create_validator_with_nominators<T: Config>(
assert_ne!(Validators::<T>::count(), 0);
assert_eq!(Nominators::<T>::count(), original_nominator_count + nominators.len() as u32);

let mut reward_map = BoundedBTreeMap::new();
for (validator, reward) in points_individual {
assert_ok!(reward_map.try_insert(validator, reward));
}
// Give Era Points
let reward = EraRewardPoints::<T::AccountId> {
let reward = EraRewardPoints::<
T::AccountId,
<T::ElectionProvider as ElectionProviderBase>::MaxWinners,
> {
total: points_total,
individual: points_individual.into_iter().collect(),
individual: reward_map,
};

let current_era = CurrentEra::<T>::get().unwrap();
Expand Down Expand Up @@ -598,8 +608,8 @@ mod benchmarks {
}

#[benchmark]
// Worst case scenario, the list of invulnerables is very long.
fn set_invulnerables(v: Linear<0, { MaxValidators::<T>::get() }>) {
// Worst case scenario according to the invulnerables limit.
gui1117 marked this conversation as resolved.
Show resolved Hide resolved
fn set_invulnerables(v: Linear<0, { T::MaxInvulnerables::get() }>) {
let mut invulnerables = Vec::new();
for i in 0..v {
invulnerables.push(account("invulnerable", i, SEED));
Expand Down Expand Up @@ -865,10 +875,17 @@ mod benchmarks {
payout_calls_arg.push((validator.clone(), current_era));
}

let mut reward_map = BoundedBTreeMap::new();
for (validator, reward) in points_individual {
assert_ok!(reward_map.try_insert(validator, reward));
}
// Give Era Points
let reward = EraRewardPoints::<T::AccountId> {
let reward = EraRewardPoints::<
T::AccountId,
<T::ElectionProvider as ElectionProviderBase>::MaxWinners,
> {
total: points_total,
individual: points_individual.into_iter().collect(),
individual: reward_map,
};

ErasRewardPoints::<T>::insert(current_era, reward);
Expand Down
Loading
Loading