Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FRAME] Add paged NMap #4604

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

[FRAME] Add paged NMap #4604

wants to merge 2 commits into from

Conversation

ggwpez
Copy link
Member

@ggwpez ggwpez commented May 27, 2024

Port of paritytech/substrate#14747

This MR implements StorageList and StoragePagedNMap. Only the latter contains code, the former is an adapter type of it. Alike it should be possible to use it as StoragePagedDoubleMap and StoragePagedMap. All variants are counted, so the Counted prefix is omitted.

API

The core API of a StoragePagedNMap is given by the trait StorageKeyedList<K, V> and very similar to a paged list - just with an additional key argument.

/// Iterator for normal and draining iteration.
type Iterator: Iterator<Item = V>;

/// Append iterator for fast append operations.
type Appender: StorageListAppender<V>;

/// Number of elements in the list under `key`.
fn len(key: K) -> u64;

/// Get the elements in append order.
fn iter(key: K) -> Self::Iterator;

/// Drain the elements in append order.
fn drain(key: K) -> Self::Iterator;

/// A fast append iterator.
fn appender(key: K) -> Self::Appender;

/// Append a single element.
fn append_one<EncodeLikeValue>(key: K, item: EncodeLikeValue) { .. }

/// Append many elements.
fn append_many<EncodeLikeValue, I>(key: K, items: I) { .. }

It does currently not support partial key iterations, but i think it should be possible to add it.

Storage Layout

The StoragePagedNMap accepts a tuple of key generators in its type signature and a tuple of matching keys in every of its functions. The KeyGenerator then allows to encode that tuple of keys to a proper storage key. We use this here, denoted as KEY. Note that since a StoragePagedList is just a special instance of a NMap, the KEY can be constructed by passing () into the KeyGenerator.

The Metadata is located at twox128(PALLET_PREFIX) ++ twox128(STORAGE_PREFIX) ++ KEY ++ "meta".

Each page with index INDEX is located at: twox128(PALLET_PREFIX) ++ twox128(STORAGE_PREFIX) ++ KEY ++ "page" ++ INDEX.

It looks a bit like this, because a "Paged NMap" basically maps a key to a "Paged List":

Screenshot 2024-05-27 at 19 21 33

Metadata

The metadata of a StoragePagedList<Value> is currently set to the metadata of a StorageMap<u32, Vec<Value>>. This is correct on the storage level, since it maps the page-index (u32) to a page (Vec<Value>). But when reading the page, it is unclear to the reader what offset to use within the page to avoid reading removed values.
FRAME metadata V17 would be needed to fix this properly so that the reader is aware of the metadata field in storage.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
@ggwpez ggwpez changed the title Port paged-nmap changes [FRAME] Add paged NMap May 27, 2024
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
@paritytech-cicd-pr
Copy link

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable 3/3
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6315870

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants