IPLD merkle-path improvements

[mildred]

Improves PR #37 and replaces PR #60. The idea is that there is only one kind of merkle-paths that are not to be confused with unixfs paths. These paths are powerful enough to be able to access multiple properties in IPLD objects and resolve merkle links.

[jbenet]

let's maybe call this one object0 instead of root?

[jbenet]

@mildred thanks, this is a strong solution possibility.

(re the unixfs paths being different, /ipfs/ links on the gateway (which are sort of unixfs paths), for example, output the concatenated data of a file, instead of the raw objects, and have directory listings, and we cannot access children links of files there)

[mildred]

the unixfs paths being different

That's a part I missed and it created much confusion, sorry for that.

[jbenet]

when we say this, do we mean:

• these paths are not good to represent files because they allow access to the raw structures underneath
• or, these paths are not good to be accessed via the web or the filesystems, because there are UI problems.

i do want to be able to use these paths via HTTP to inspect the underlying data structures. but it's ok if i cannot get a proper file or directory representation out of it.

the latter part of this statement makes me somewhat more at peace with any of:

• disallowing access of link properties via the paths
• disallowing transparent dereferencing without the use of $obj/link/
• separating "the link" and "the link properties" (same as the previous one in a way)

---

i think what we really need here is more concrete datastructure examples, and see how the pathing would want us to get to the stuff. maybe this will show us which solution is the best and guide our thinking.

perhaps we could write:

• an fs example (directories, sharded files, file attributes in dir entries)
• a version control example (commit histories)
• a social network example (users, messages, relationships) (maybe something like a simple version of foaf)
• a crypto network example (keys, derived keys, signatures, certificates, attestations) (i'll write this one at least)
• a blockchain example (block histories, transactions, etc)

(any other killer use case for ipld that we could model here?)

[mildred]

What I meant is these paths are not good to represent files because they allow access to the raw structures underneath. That's why I took the analogy of the / proc filesystem which is not good for storing user files, but is still valuable as a filesystem. I might not have been clean enough, it's late at night in France and I need sleep :-)

There might be UI problems as the . separator is not understood as such by browsers for example. I don't think this will be a real problem though.

[jbenet]

That's a part I missed and it created much confusion, sorry for that.

not at all, they look the same right now, which is definitely confusing. we could:

• leave /ipfs/... for unixfs
• use /ipld/... for raw ipld data.

not sure. in the end i still have trouble with the unixfs--ipld dichotomy. im not sure how to get unixfs to play well with other things (like commits). it is likely that:

• we will have a 1 + N datastructure and 1 + N path world (1 for ipld and N for every datastructure)
• or a 1 + N datastructure but 1 + 1 path world (1 for ipld and 1 for the derived datastructs together).

the latter is hard but if we can get it it would be much easier to reason about. will be very difficult to think of different datastruct-specific paths :/

[jbenet]

i do indeed recognize that there is value to try to get everything to be one path system, for example, give up on link properties and make this:

{
  "cat.jpg": {"link": "Qmcatjpg..."},
  "foo": {"link": "Qmfoo..."},
  "@attrs": {
    "cat.jpg": {
      "mode": 777,
      "owner": "jbenet"
    }
  }
}

://////

[mildred]

Your example works well because it represents a file structure that is well represented with unix-like paths. But what if we try to represent anything else. What if keys contain binary data ? You need an escape mechanism.

Or perhaps I'm not thinking strainght as i am tried. In any case, the path definition is not impacting (or should not impact) the IPLD format in itself. So perhaps that's an issue that can be resolved later?

[mildred]

I rephrased the paragraph you commented here to remove ambiguities. You talked about examples, but they are already there at the end of ipld.md.

If you want to unify every paths under the same prefix, how would that work in practice? For example, how will /ipns/ be managed (or am I lagging behind old concepts)?

[mildred]

{
  "cat.jpg": {"link": "Qmcatjpg..."},
  "foo": {"link": "Qmfoo..."},
  "@attrs": {
    "cat.jpg": {
      "mode": 777,
      "owner": "jbenet"
    }
  }
}

here, you won't be able to access both @attrs and cat.jpg using the same path mechanism, unless you start telling people they can't name a file @attrs.

I don't think you can force people to use the same path for everything. It's not practical. Some applications might need some things not provided by default.

[jbenet]

@mildred I've made https://github.com/ipfs/ipld-examples to try and resolve this question. So far i've only made two examples unixfs and post. take a look, i changed things up somewhat after giving more thought. I'm still not decided, but i'm seeing how horrible some of those options are.

could you please double check my work? im not sure i got everything right-- I may have messed up some of them, given that there are things you mentioned were problems but i didnt run head into them. (escaping . when using it as a delimiter, for example. may be that this could be a non issue, but anyway, its separate).

I think so far, my favorites are (8), (4), and (5). -- the other feel odd or are very, very confusing.

Also, see the unixfs pathing-- i made it a separate thing there too, and i am more convinced this is the right thing to do.

• https://github.com/ipfs/ipld-examples/tree/master/post
• https://github.com/ipfs/ipld-examples/tree/master/unixfs

[drvirgilio]

I think . should not be used to do traversal. I think \. should. This way there is no need to escape for key names such as notes.txt which I think would occur more often.

[mildred]

I updated this branch to ipld-spec and added my understanding of the path mechanism (8)

[jbenet]

I think we should ditch this paragraph -- it's not totally accurate, FSes can be implemented, just the hoops get ugly (dirA/@link/dirB/@link/fileC/data), and we want something cleaner. I think the unixfs spec on top of IPLD can discuss the choices there.

[mildred]

Ok, as long as I can store a file named @link on my unixfs filesystem (which would at one point in some future would become my root filesystem I'd imagine).

[jbenet]

i think we can keep this, and it expresses enough.

[mildred]

Right

[jbenet]

@mildred things look good. I'm good to leave the cases there for now, and decide on the lenient vs strict down the road. we can implement strict for now and see how it goes?

[jbenet]

I'm good to merge as is, and continue from there.

[jbenet]

@mildred lmk if you are ready too, and i'll merge. else what else to do?

[mildred]

I'm ok with this as well.