Multicodec

[jbenet]

This PR rebases ipld on top of multicodec.

See also: multiformats/go-multicodec#6

---

Changes:

3b95669 (Juan Batiz-Benet, 3 minutes ago)

 rm dag builder thing

c3a7ff9 (Juan Batiz-Benet, 5 minutes ago)

 rewrote coding in terms of multicodec

60e3b77 (Juan Batiz-Benet, 18 minutes ago)

 added pb for decoding old protobufs

[jbenet]

This doesn't yet introduce the changes @diasdavid @mildred and I discussed. follow-on PR.

@diasdavid can you tag me whenever you push support as we discussed in node-ipld?

[daviddias]

Sure, currently I've my node-ipld thing here https://github.com/diasdavid/node-ipld working with the 'mvp ld' we talked to jump in to IPRS records land.

[mildred]

@jbenet I looked up this PR, nice work. There is something I don't understand though. When you transform protobuf to JSON, you construct a JSON like this one:

{
  links: [
    {hash: "...", name: "...", size: 0},
    ...
  ],
  data: "..."
}

This is however very different from other examples you gave when designing IPLD:

{
  "@context": "/ipfs/<hash-of-schema>/unixdir",
  "<filename1>": { "@value": "<hash1>", "unixMode": <mode1> },
  "<filename2>": { "@value": "<hash2>", "unixMode": <mode2> },
}

Do you agree with me that the code of IPFS must read this JSON in some defined way (the flattening algorithm) to construct a list of links, each having a name, and pointing to another IPFS object.

I understand well how this algorithm will work in the second snippet above, the link name being the keys of the JSON. But for the first example, which is converted from protobuf, how will the algorithm will be able to construct the list of links in a consistent way?

I would have in fact imagined that the protobuf objects would convert to something like this:

{
  @context: <link to protobuf context>
  @value: <data segment>
  <link name>: { hash: <hash>, size: <size> },
  <link name>: { hash: <hash>, size: <size> },
  // "@" characters are escaped as "\@" (and "\" as "\\")
  ...
}

Why didn't you choose this format ?

[jbenet]

@mildred this is specifically the OLD (current) protobuf format. link names are not unique currently, and there is the "data" segment. it follows the proto there. i wish we could transparently convert to the new style of doing things but dont think we can. we must preserve backwards compat to avoid breaking links.

what we can do though is add "mlink" and so on there.

[mildred]

The main question is how the flattening algorithm that will be used to determine link names will work under this old format. Either:

• we transform the flattening algorithm to use the name attribute when provided instead of the flattened key of the link in the JSON. In case some links appear twice, we must define a process order.
• we convert to another format that contain the link name as key, but link names are not unique. We must decide if we take the first link of that name, or the last (whatever IPFS does already). Duplicate links could be preserved in a separate way.