IPFS Repo Feature - Implement locks

[daviddias]

IPFS implementations may use multiple processes, or may disallow multiple processes from using the same repo simultaneously. Others may disallow using the same repo but may allow sharing datastores simultaneously. This synchronization is accomplished via locks.

Full information: https://github.com/ipfs/specs/blob/fix-repo/repo/README.md#locks
Complete Repo Spec - https://github.com/ipfs/specs/blob/fix-repo/repo/README.md

[daviddias]

I'm thinking that it would be nice to have a store, that builds on top of abstract-blob-store, introduces features as locks, but doesn't necessarily require for every blob-store implementation to be crafted by hand.

This will require things like upgrading lockfile to understand the abstract-blob-store

[daviddias]

@whyrusleeping just to make sure I get this right, it is supposed for ipfs-repo to support more than one type of store at the same time? Looking at a regular ipfs repo, seems like datastore is using levelDB and the rest is just straight fs. (What I'm trying to understand is if I can have a clean interface for this things so that it doesn't get super complicated to make it work in the browser as well)

[daviddias]

Ok, going to change how ipfs-repo is changed a bit, so that we support multiple types of storage mechanism depending on the type of data being stored.

I'm going to follow the diagram:

With this, we can select the best datastore for each type of data, while still keeping the option of using levelDB/IndexedDB without any big changes, so it works nicely in the browser

• https://github.com/maxogden/abstract-blob-store
• https://github.com/mafintosh/fs-blob-store
• https://github.com/substack/idb-blob-store

[jbenet]

Some clarity on the question about stores:

go-ipfs history:

• we started with leveldb
• we moved blocks to flat-fs
• we are in process of moving things off leveldb entirely (like pinset in dev0.4.0)
• my goal is to get rid of leveldb entirely

regardless, it is the case that there is "one datastore", it's just split in two underneath the hood:

• https://github.com/ipfs/go-ipfs/blob/master/repo/fsrepo/fsrepo.go#L97
• https://github.com/ipfs/go-ipfs/blob/master/repo/fsrepo/fsrepo.go#L381-L390

so:

• as far as "the repo" is concerned, there's one datastore.
• the go-ipfs fs-repo implementation uses one "mount datastore" with two on-disk datastores. this will move to just one on-disk datastore.
• js-ipfs could get away with going straight for the "just one store". the pinset in dev0.4.0 was the hardest part to move good.

I can go through and look at what's left in leveldb, and see about moving it off. i think it may just be dht stuff at this point. (which should be objects)

[jbenet]

And your diagram LGTM! (nice colors!) -- it does not mean changes on fs-repo spec or go-ipfs, right? only on how it's accessed in js-ipfs?

[daviddias]

Nice! Thank you for bringing this clarifications into the thread, I didn't know we were moving away from two data stores.

The diagram doesn't propose any change to go-ipfs, it is just a way to deal with the current multi datastore current and enable for these to be replaced by IndexedDB so that it works in the browser.

I've been using a dev0.4.0 repo to create my tests and there is indeed levelDB stuff, I can avoid trying to parse it for now, but is it official that 0.4.0 won't have any more levelDB goodness? @jbenet @whyrusleeping

Also, I'm curious to learn why we want to remove levelDB entirely over flatfs.

[daviddias]

-> locks + version with locks -> https://github.com/ipfs/js-ipfs-repo/pulls

[masylum]

I like the direction. 👍 @diasdavid

[jbenet]

but is it official that 0.4.0 won't have any more levelDB goodness? @jbenet @whyrusleeping

dev0.4.0 wont be out of leveldb yet. if we push for it we wont ship it before xmas, and we should. that can come later. repo migrations have worked very well and not been very problematic so far. (caveat: we need to do some user research on this, with people building apps which compose go-ipfs)

[daviddias]

I've updated the README with more documentation, architecture.

locks work, it has tests and everything, closing this issue :)