Commentary on v2 Storage & Distribution Runtime Specification

[bedeho]

Background

A specification was recently prepared for the v2 storage and distribution system, and this comment is an attempt to summarise some observations on an alternative path not taken, in order to bring greater clarity around why the specification looks as it does.

Commentary

The conceptual role of the storage and distribution system is to maintain the following state

What data objects exist, and for each, who owns, stores and distributes it

Storage vs. Ledger

Importantly, this information is kept in the storage (state) of the chain, not just in the ledger. A ledger based data directory would simply do authentication of actors and retain and index of data objects and corresponding owner. It would otherwise ignore semantics of all operations. The runtime could itself do automatic assignment of liaisons storage providers and even distributors, however, this assignment would not be sensitive to the storage utilisation of providers chosen, as the chain does not know who is doing what at any time. The actual processing of the operations would have to be done off-chain, using something like the query node, and it would check that the operations where feasible.

Runtime Incentives

If we had a system where the chain itself was responsible for directly auditing the services rendered by providers, for example through probabilistic challenge response audits that can lead to slashing, then it this state would have had to live in the chain state, but we currently don't.

Bags

The very notion of bags indeed a result of needing to more succinct identifiers for collections of providers, in terms of their obligations, in order to economise on storage access frequency. If this representation was not in storage, then the need for bags would also go away. You would just have totally flexible assignment of providers to individual objects. In theory, you may end up running into limits of block space itself if you have a large data directory with fast dynamics, which then could require introduction of bags again.

Analysis

Obviously, a ledger based approach would lessen the constraints, such as the number of storage and distributor buckets there can be. It would also allow updating policies for automatically assigning providers to bags without doing runtime upgrades. In general updating the system in other ways would be easier. The cons is that its more complex, and more or less permanently rules out any more sophisticated on-chain auditing.