feat: remove fixed shards #9219

wacban · 2023-06-19T17:24:45Z

In this PR I'm removing the fixed shards field from the shard layout. I removed the fixed_shards field directly from the ShardLayoutV1. This implementation is quite hacky as it does not follow the typical flow for making protocol changes. This approach needs to be discussed and agreed on first but here is an implementation just to get a feel for it.

The motivation for removing fixed shards is that fixes_shards break the account ids contiguity within a shard. A sub-account of a fixed shard is assigned to the same shard as the fixed account but it may fall in the middle of account range of another shard. For example let's consider the following shard layout:

fixed_shards = ['fixed']
boundary_accounts = ['middle']

In this shard layout we have three shards:

0 - 'fixed' and all sub-accounts
1 - [..'middle'] with exception for any sub-account of fixed
2 - ['middle'..] with exception for any sub-account of fixed

Because of the fixed account, shards 1 and 2 are now full of holes for any subaccounts of 'fixed'. This property of fixed_shards makes it hard to perform any range operations on a shard such as resharding or deletion.

The proper way to do it would be to add ShardLayoutV2 - a copy of V1 with fixed_shards removed and version bumped. Due to the way that we use for storing state date in storage - using the shard_uid as a prefix for the storage key - this approach would require a careful migration. One way to go about it would be to perform a null-resharding in epoch preceeding the shard layout version bump. There the node would have to create a copy of the state and store it in storage with the new version in shard_uid. Once the state is duplicated the node can keep applying changes to both states, same as in resharding. Such a migration would need to be prepared very carefully and would take a lot of time and effort.

The reason why I believe that we can get away without new shard layout version is that fixed_shards are not used in production and the shard version is not actually stored in the state. The ShardLayoutV1 with empty fixed_shards is semantically equivalent to ShardLayoutV1 with the fixed_shards field removed.

It's worth mentioning that the migration pains would be removed if we decided to change the storage structure to one that doesn't have the shard_uid in the storage key.

wacban · 2023-06-19T17:36:13Z

the next steps are:

discuss this approach with the team
- also think if this requires a NEP despite technically not being a protocol change
find any remaining tests (nayduck) that depend on fixed_shards and fix them
test that everything works well in testnet and mainnet

wacban · 2023-06-20T12:53:45Z

surprisingly nayduck doesn't seem to have a dependency on fixed_shards
https://nayduck.near.org/#/run/3108

wacban · 2023-06-22T10:38:06Z

Discussed on zulip here and we're good to go.

shreyan-gupta

Looks good!

core/primitives/src/shard_layout.rs

integration-tests/src/tests/client/flat_storage.rs

wacban · 2023-06-29T13:22:52Z

note to self: describe the change in changelog

pugachAG

LGTM!

pugachAG · 2023-06-29T14:06:59Z

core/primitives/src/shard_layout.rs

+            // Note: As we scale up the number of shards we can consider
+            // changing this method to do a binary search rather than linear
+            // scan. For the time being, with only 4 shards, this is perfectly fine.
+            let mut shard_id = 0 as ShardId;


super nit: this looks a bit weird, let mut shard_id: ShardId = 0 seems more canonical to me

In this PR I'm removing the fixed shards field from the shard layout. I removed the fixed_shards field directly from the ShardLayoutV1. This implementation is quite hacky as it does not follow the typical flow for making protocol changes. This approach needs to be discussed and agreed on first but here is an implementation just to get a feel for it. The motivation for removing fixed shards is that fixes_shards break the account ids contiguity within a shard. A sub-account of a fixed shard is assigned to the same shard as the fixed account but it may fall in the middle of account range of another shard. For example let's consider the following shard layout: ``` fixed_shards = ['fixed'] boundary_accounts = ['middle'] ``` In this shard layout we have three shards: - 0 - 'fixed' and all sub-accounts - 1 - [..'middle'] with exception for any sub-account of fixed - 2 - ['middle'..] with exception for any sub-account of fixed Because of the fixed account, shards 1 and 2 are now full of holes for any subaccounts of 'fixed'. This property of fixed_shards makes it hard to perform any range operations on a shard such as resharding or deletion. The proper way to do it would be to add ShardLayoutV2 - a copy of V1 with fixed_shards removed and version bumped. Due to the way that we use for storing state date in storage - using the shard_uid as a prefix for the storage key - this approach would require a careful migration. One way to go about it would be to perform a null-resharding in epoch preceeding the shard layout version bump. There the node would have to create a copy of the state and store it in storage with the new version in shard_uid. Once the state is duplicated the node can keep applying changes to both states, same as in resharding. Such a migration would need to be prepared very carefully and would take a lot of time and effort. The reason why I believe that we can get away without new shard layout version is that fixed_shards are not used in production and the shard version is not actually stored in the state. The ShardLayoutV1 with empty fixed_shards is semantically equivalent to ShardLayoutV1 with the fixed_shards field removed. It's worth mentioning that the migration pains would be removed if we decided to change the storage structure to one that doesn't have the shard_uid in the storage key.

* fix(state-sync): finalize using flat storage * fix(state-sync): Applied values from state parts should be inlined when applicable (#9265) * fix(state-sync): Applied values from state parts should be inlined when applicable * fix(state-sync): Applied values from state parts should be inlined when applicable * fix(state-sync): Applied values from state parts should be inlined when applicable * fix(state-sync): Applied values from state parts should be inlined when applicable * clippy * fmt * logging * scan from a hash * refactor: add FlatStateValue constructor for on-disk values (#9269) This logic is used in multiple places, so it makes sense to abstract it behind a method in `FlatStateValue`. `on_disk` name is chosen over `new` since this logic is only applicable for values to be persisted on disk. Later we will also add inlining for values as part of Flat Storage Delta and those will have lower inlining threshold, so we will have another constructor. * refactor: Move adjust-db tool to database tool (#9264) Refactoring of `adjust-db` tool, moving it to the `database` toolset where there's of database tools should live. * feat: remove fixed shards (#9219) In this PR I'm removing the fixed shards field from the shard layout. I removed the fixed_shards field directly from the ShardLayoutV1. This implementation is quite hacky as it does not follow the typical flow for making protocol changes. This approach needs to be discussed and agreed on first but here is an implementation just to get a feel for it. The motivation for removing fixed shards is that fixes_shards break the account ids contiguity within a shard. A sub-account of a fixed shard is assigned to the same shard as the fixed account but it may fall in the middle of account range of another shard. For example let's consider the following shard layout: ``` fixed_shards = ['fixed'] boundary_accounts = ['middle'] ``` In this shard layout we have three shards: - 0 - 'fixed' and all sub-accounts - 1 - [..'middle'] with exception for any sub-account of fixed - 2 - ['middle'..] with exception for any sub-account of fixed Because of the fixed account, shards 1 and 2 are now full of holes for any subaccounts of 'fixed'. This property of fixed_shards makes it hard to perform any range operations on a shard such as resharding or deletion. The proper way to do it would be to add ShardLayoutV2 - a copy of V1 with fixed_shards removed and version bumped. Due to the way that we use for storing state date in storage - using the shard_uid as a prefix for the storage key - this approach would require a careful migration. One way to go about it would be to perform a null-resharding in epoch preceeding the shard layout version bump. There the node would have to create a copy of the state and store it in storage with the new version in shard_uid. Once the state is duplicated the node can keep applying changes to both states, same as in resharding. Such a migration would need to be prepared very carefully and would take a lot of time and effort. The reason why I believe that we can get away without new shard layout version is that fixed_shards are not used in production and the shard version is not actually stored in the state. The ShardLayoutV1 with empty fixed_shards is semantically equivalent to ShardLayoutV1 with the fixed_shards field removed. It's worth mentioning that the migration pains would be removed if we decided to change the storage structure to one that doesn't have the shard_uid in the storage key. * remove near-cache dep from near-vm-runner (#9273) Part of #8197 * fix(state-sync): Clear flat storage outside of ClientActor (#9268) Deleting multiple GBs from RocksDB is definitely taking more than 100ms. In case a node catches up, it can potentially make a chunk producer or a block producer miss its chunk/block. This PR moves deletion of Flat State to `SyncJobsActor` to avoid blocking `ClientActor`. * remove near-stable-hasher dep from near-vm-runner (#9270) Part of #8197 * fix(state-sync): Applied values from state parts should be inlined when applicable (#9265) * fix(state-sync): Applied values from state parts should be inlined when applicable * fix(state-sync): Applied values from state parts should be inlined when applicable * fix(state-sync): Applied values from state parts should be inlined when applicable * fix(state-sync): Applied values from state parts should be inlined when applicable * clippy * fmt * fix(state-sync): Finalize state sync using flat storage * fix(state-sync): Finalize state sync using flat storage --------- Co-authored-by: Anton Puhach <anton@near.org> Co-authored-by: Jure Bajic <jure@near.org> Co-authored-by: wacban <wacban@users.noreply.github.com> Co-authored-by: Ekleog-NEAR <96595974+Ekleog-NEAR@users.noreply.github.com> Co-authored-by: near-bulldozer[bot] <73298989+near-bulldozer[bot]@users.noreply.github.com>

wacban requested review from pugachAG, Longarithm, shreyan-gupta and robin-near June 19, 2023 17:24

wacban marked this pull request as ready for review June 22, 2023 10:38

wacban requested a review from a team as a code owner June 22, 2023 10:38

wacban linked an issue Jun 23, 2023 that may be closed by this pull request

[Resharding] Remove the fixed shards field from the shard layout #9202

Closed

shreyan-gupta approved these changes Jun 27, 2023

View reviewed changes

core/primitives/src/shard_layout.rs Show resolved Hide resolved

integration-tests/src/tests/client/flat_storage.rs Show resolved Hide resolved

wacban force-pushed the waclaw-remove-fixed-shards-hacky branch from 74f7238 to f65557d Compare June 28, 2023 11:39

Longarithm approved these changes Jun 29, 2023

View reviewed changes

pugachAG approved these changes Jun 29, 2023

View reviewed changes

wacban added 6 commits June 30, 2023 11:28

feat: remove fixed shards from shard layout

2fdc2d9

fix some tests

8b62d78

self review

9e11057

fix test

b331fea

add backwards compatibility test

d4cb9f2

changelog

2d99f84

wacban force-pushed the waclaw-remove-fixed-shards-hacky branch from 2c253d1 to 2d99f84 Compare June 30, 2023 10:31

super nit

d610d5a

wacban added the S-automerge label Jun 30, 2023

Merge master into waclaw-remove-fixed-shards-hacky

f5bedab

near-bulldozer bot merged commit a64668c into master Jun 30, 2023

near-bulldozer bot deleted the waclaw-remove-fixed-shards-hacky branch June 30, 2023 10:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: remove fixed shards #9219

feat: remove fixed shards #9219

wacban commented Jun 19, 2023 •

edited

Loading

wacban commented Jun 19, 2023

wacban commented Jun 20, 2023

wacban commented Jun 22, 2023

shreyan-gupta left a comment

wacban commented Jun 29, 2023

pugachAG left a comment

pugachAG Jun 29, 2023

feat: remove fixed shards #9219

feat: remove fixed shards #9219

Conversation

wacban commented Jun 19, 2023 • edited Loading

wacban commented Jun 19, 2023

wacban commented Jun 20, 2023

wacban commented Jun 22, 2023

shreyan-gupta left a comment

Choose a reason for hiding this comment

wacban commented Jun 29, 2023

pugachAG left a comment

Choose a reason for hiding this comment

pugachAG Jun 29, 2023

Choose a reason for hiding this comment

wacban commented Jun 19, 2023 •

edited

Loading