Typed Storage Keys

[tadeohepperle]

Fixes #1201

Background Info

Previously, static storage keys were represented as a Static<Encoded> (wrapper around a Vec<u8>) that is created from a concrete type T: Encode by scale encoding it.
Previously the storage::Address struct was generic over the type of key: K, and held a Vec<K> as a value. This K was Static<Encoded> for static storage keys and scale_value::Value for dynamic storage keys.

What changed

For static storage keys, used in the codegen, we add a new type StorageKey<K> that is just a value of K scale encoded into bytes, but keeps the type information of K in a PhantomData field.

A new trait StorageMultiKey is introduced, that can be implemented by anything that represents one or multiple storage keys:

pub trait StorageMultiKey {
    fn keys_iter(&self) -> impl ExactSizeIterator<Item = &dyn EncodeAsType>;

    // Skips over the first 16 bytes of the storage address (pallet hash + entry hash) and then tries to
    // reconstruct the StorageMultiKey from the sequence of hashes (only possible if all hashes are concat hashes)
    fn decode_from_address_bytes(
        address_bytes: &[u8],
        hashers_and_ty_ids: &[(StorageHasher, u32)],
        metadata: &Metadata,
    ) -> Option<Result<Self, Error>>;
}

This trait is then implemented for:

• StorageKey<K>,
• tuples of storage keys: (StorageKey<A>, StorageKey<B>, ...)
• Vec<scale_value::Value>

The Address struct representing a storage address is now generic over a Keys: StorageMultiKey type, instead of holding a Vec.

This allows us to construct storage addresses that use tuples of storage keys (StorageKey<A>, StorageKey<B>, ...) in the generated code.

Also when getting StorageResponse's back from the Backend, we can now attempt to reconstruct the respective StorageMultiKey from the address bytes.

[jsdw]

Ah ok, so while &dyn EncodeAsType won't work any more when I make EncodeAsType non-object-safe (ie generic over TypeResolvers), since Subxt will probably continue to use concrete types like PortableRegistry instead of relying too much on TypeResolver, it'll be easy enough to make a wrapper trait that is object safe in the same way I had to in scale-encode :)

[lexnv]

s/construc/construct

[lexnv]

dq: Unfinished doc

[lexnv]

Nit: I would add a comment that for a valid storage key, we'd need at least 8 bytes for the pallet hash and 8 for the entry

[lexnv]

I would add a small section about keys needing at least 16 bytes (pallet ++ entry)

Edit: ah this can also mean we are needing 8 bytes and so on (prob good to add a u8 here?)

[lexnv]

byt a concat?

[niklasad1]

this may end up with an empty slice if address_bytes.len == 16 ok?

[tadeohepperle]

Yes, that is true, but it is okay I think.

[jsdw]

Yeah I think that's ok; if it matters (ie we expect a non-empty hash next) then the next thing will error instead

[niklasad1]

I find this is a bit hard to hard to read, maybe we could do contains_key, contains_plaintext or is_concat_with_key... not sure

[jsdw]

I chanegd it from contains_key or something I think! I went with ends_ just to fit in more nicely with the above method which is giving the length of the hash apart from the key bit.

contains_key would work too though for me!

[lexnv]

Either looks good to me :D

[jsdw]

Should this be a part of the public API? Maybe it could just be a small internal helper fn instead :)

[jsdw]

Nit: reset() type functions on iterators feel a bit unidiomatic to me (and mutating an iterator scares me a little). Not really a big deal but we could be more idiomatic perhaps by just making the iteration be a separate thing like so:

// rename `StorageHAshersIter` to this but keep methods the same:
pub struct StorageHashers { 
  // same methods.. 
 
    pub fn iter(&'_ self) -> StorageHashersIter<'_> {
        StorageHashersIter { hashers: self, idx: 0 }
    }
}

// impl `Iterator` on a separate type instead (then we can iterate over them 
// as many times as we want and no iterators are being mutated. Maybe a bit like:
pub struct StorageHashersIter<'a> {
    idx: usize,
    hashers: &'a StorageHashers
}

impl <'a> Iterator for StorageHashersIter<'a> {
    fn next(&mut self) {
        let item = self.hashers.hashers_and_ty_ids.get(self.idx).copied()?;
        self.idx += 1;
        Some(item)
    }
}

[jsdw]

Maybe this is a bit of a pain though, and if so I'm happy to stick with the current approach!

[tadeohepperle]

Oh yeah, the reset is a bit ugly, I implemented a seperate StorageHashersIter<'a> struct now.

[jsdw]

This looks great to me now; just a couple of very small suggestions! Great job @tadeohepperle!

[jsdw]

Intentional? :)

[jsdw]

One println!() that I think is unintended but otherwise looks awesome :)

[lexnv]

Nice! 👍

[lexnv]

Nice job here 👍