Sharding IPLD objects

[jbenet]

This note gathers the constraints + will drive toward a design of object sharding in IPFS and IPLD. Object sharding is the algorithms and formats used to represent a single (virtual) large object out of many smaller ones. Think of this like the way large directories are represented in modern filesystems (RB-Trees, B-Trees, HTrees, etc).

Sharding IPLD objects in general is a useful thing. instead of implementing it for unixfs and other datastructs each time, we could implement it once. it could be a datastruct the others employ, or maybe -- if it is simple enough -- it belongs as part of IPLD itself.

Constraints to support:

• efficient in the small case (1 to 5 nodes)
• allows user-chosen sharding (eg for small numbers of nodes, may want specific construction)
• large fanouts (millions or billions)
• efficient access
• minimize insertion re-writes (shadowing/cloning)
• upgradeable algorithms (can signal which sharding algo via version, or even with a key/val)
• union style fanouts
• hierarchical style fanouts (patricia tries)

For large fanouts, look at

• all filesystems research into indirect block topologies
• ext4 dir entries (list + tree) https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Directory_Entries
• HTree https://en.wikipedia.org/wiki/HTree
• PHTree http://phunq.net/pipermail/tux3/2013-January/000026.html
• B-trees + shadowing + clones http://liw.fi/larch/ohad-btrees-shadowing-clones.pdf
• djb's cdb http://cr.yp.to/cdb.html
• @tv42's set + multiset on ipfs (used already for 0.4.0+ pinset) pinning index (as ipfs objects) and cdb discussion #4 (comment)

---

case for supporting it on-top of IPLD

• It is nice that the IPLD spec is very simple. Finding a nice way to support this without complicating it much will be hard-- the constraints above do not bode well for this.
• can define it as a different datastruct, should not be hard for other datastructs to extend it
• flexible algorithms for sharding may complicate IPLD

case for supporting it in IPLD

• we could have a very powerful datastructure if sharding came everywhere
• merkle-linking in IPLD is already like hierarchical fanout sharding of a single massive tree, this is just sharding within a single level.
• IPLD already has flexible algos in multicodec
• could use a directive like @shard or something
• could be an IPLD extension if not properly in core spec.

---

cc @whyrusleeping @lgierth @diasdavid @cryptix @ion1 @mildred @tv42 @wking

[ion1]

It might be useful to take advantage of chunking by a rolling hash with large flat data structures receiving small arbitrary changes, such as a directory.

[davidar]

Radix trees have the benefit of being deterministic (invariant to the order of insertions), which helps with deduplication.

Edit: also see http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.37.5452

[jbenet]

cc @whyrusleeping comment here.

[jbenet]

From the IPLD spec, one very trivial but extensible thing to do all-IPLD-object-wide is something like this:

{
  "@extends": [
    "/ipfs/<hash1>",
    "/ipfs/<hash2>",
    "/ipfs/<hash3>",
  ],
  ... // more properties
}

which would merge the objects like so:

o1 = get(/ipfs/<hash1>)
o2 = get(/ipfs/<hash2>)
o3 = get(/ipfs/<hash3>)
o4 = get(/ipfs/<hash4>) // our object above

merged = _.extends(o1, o2, o3, o4)

meaning that we start with the first, and we patch in order. so a property in o4 replaces those preceding it. this is very simple and can likely be used to support the desires mentioned earlier in this issue (more complicated structures).

---

It would be useful to have examples of directory sharding with:

• B-trees + shadowing + clones
• a Radix tree

expressed as the corresponding JSON or YML syntax for IPLD. (i use JSON above because it tends to be clearer, but i prefer YML). that way we can see what it takes to support it natively in IPLD.

[davidar]

Personally I'd like it if this was a core part of IPLD, so that large objects were transparently sharded without the user having to explicitly do anything (much like the transparent merkle-linking). Here's an example of a manual radix tree though:

r:
  om:
    an:
      e: 1
      us: 2
    ulus: 3
  ub:
    e:
      ns: 4
      r: 5
    ic:
      on: 6
      undus: 7

Sub-trees can be split into separate (merkle-linked) objects, analogously to what we already do with chunking linear streams.

[jbenet]

• also https://en.wikipedia.org/wiki/HAT-trie

[mildred]

I think this should not be implemented below the basic IPLD data model: you want, depending on the application, to control where to split objects. For example, metadata should be easily accessible while the complete file data could take a little more time to load.

I think this should be an extension of IPLD. And you could have the choice to:

• fetch only the requested object
• fetch the logical record composed of multiple objects reassembled into a bigger one

It might be necessary to think how exactly we are going to reassemble the objects. Which algorithms. In particular. Especially considering that's not just about tree merging, but also data string merging.

For example, an object might be composed of a huge binary data string (for example the object representing a huge file). This data string shoud be able to be split among different physical IPLD objects.

The way I would implement it is to have index objects that links to sub parts, and specify with the link the way they are merged. Something like jbenet suggested

@davidar I disagree with you when you say that it should be completely transparent. The application should have control over the splitting of files. We could have a library that does a good job of splitting when the application author doesn't want to bother, but we should be able to have full control.

[jbenet]

Agree in full with @mildred

[jbenet]

Some links for references.

• https://en.wikipedia.org/wiki/Hash_array_mapped_trie
• search the persistent data struct lit more.
• http://www.staff.city.ac.uk/~ross/papers/FingerTree.html
• I started making a "recursive hash table", that's similar to typical associative array optimizations.
• at the extreme of that lie thigns like Judy Arrays (tries)
• And ofc, I keep returning to finger trees, and patricia tries.
• https://infoscience.epfl.ch/record/64398/files/idealhashtrees.pdf

A tricky problem here is tuning the trade off between "good datastructures that create very small nodes" and "good sized objects to minimize the hashing costs, cache misses, and network ops". Basically, there's sharding, which represents one object with many small ones. and then there's aggregation, to represent many small objects with a big one. Most good approaches to sharding from data struct literature yield excellent datastructs with many small objects, which we'd want to aggregate. aggregation is tricky, because making it stable (convergent, for hashing) and efficient (minimize update costs) is challenging. Aggregation is not straight forward; because "bundling many objects in the underlying transport instead of in the data model" is not enough, as that does not minimize hashing and random access advertisements.

[jbenet]

More on HAMTs

• https://infoscience.epfl.ch/record/64398/files/idealhashtrees.pdf
• http://michael.steindorfer.name/publications/oopsla15.pdf
• http://bendyworks.com/leveling-clojures-hash-maps/ (links to impls)
• https://github.com/facebook/immutable-js/blob/master/src/Map.js

[whyrusleeping]

@jbenet patricia trees would be cool. But figuring out how to aggregate them in a way that is stable after random insertion orders is going to be very tricky.

The HAMT sounds a lot like what you were hacking on in SF, except for the part about counting bits for non-nil pointers. With that trick, we avoid having hugely bloated nodes which was my primary complaint with your previous idea.