track: IPFS Repo

[daviddias]

IPFS Repo is the storage driver for IPFS.

To learn more about the repo, you can

• Read the IPFS Repo Spec https://github.com/ipfs/specs/tree/master/repo

Implementation Roadmap

• IPFS Repo JS Implementation https://github.com/ipfs/js-ipfs-repo
• Integration of BlockService with IPFS Repo https://github.com/ipfs/js-ipfs-blocks
• Integration of IPFS Repo with js-ipfs https://github.com/ipfs/js-ipfs/blob/master/src/ipfs-core/index.js
• jsipfs repo init Not used anymore
  • core
  • http-api
  • cli
• jsipfs repo stat
  • core
  • http-api
  • cli
• jsipfs repo gc {difficulty: moderate} requires storage of refs (track: Pinning #59)
  • core
  • http-api
  • cli

[daviddias]

Updated this issue

[hackergrrl]

Hey @diasdavid. I have some questions about js-ipfs repo init that I could use some clarification on:

1. What specifically does js-ipfs repo init need to do? I'm unclear on where repo init differs from ipfs init in this case. Do things like key generation happen here, or is that a separate non-repo init step? Some help on understanding the boundaries between repo initialization and general node init would be appreciated.
2. According to the current API the user must specify an FS path to the IPFSRepo constructor: doesn't this lock the js-ipfs repo to the fs-repo implementation? What if I want an in-memory repo or something on S3?

Thanks!

[daviddias]

Starting from the bottom:

According to the current API the user must specify an FS path to the IPFSRepo constructor: doesn't this lock the js-ipfs repo to the fs-repo implementation? What if I want an in-memory repo or something on S3?

You can find on the Architecture diagram on the README of js-ipfs-repo that an IPFS repo is constructed by several 'subrepos' or blob-store instances, to enable the user to have a repo running on fs, an in memory db, indexeddb, even a networkDB or mixed :)

The benefit of using abstract-blob-store is that paths are handled as keys and the blob store implementation deals with them. An fs-blob-store will deal with the key as a path, an level-blob-store will handle the key as a key. You can find other varieties here https://github.com/maxogden/abstract-blob-store#some-modules-that-use-this

In fact, you can see that js-ipfs-repo already has tests that run (with locks included) both in Node.js (with fs-blob-store) and in the browser (with indexedDB-blob-store) https://github.com/ipfs/js-ipfs-repo/tree/master/tests.

What specifically does js-ipfs repo init need to do? I'm unclear on where repo init differs from ipfs init in this case. Do things like key generation happen here, or is that a separate non-repo init step? Some help on understanding the boundaries between repo initialization and general node init would be appreciated.

repo init needs to ensure that a repo can be mounted with that base path/key and the blob-stores passed (making sure a repo doesn't exist there yet, for example)

ipfs init, will run repo init and then generate the peer Id (with respective private and pub key) and store the config object, the default blocks on the blockstore, etc.

I hope this makes it more clear, let me know if there are still questions :)

[hackergrrl]

That helps, @diasdavid.

Doesn't this mean that all of the repo's stores must share the same key? What if that doesn't make sense in all implementations? E.g. I wish to store keys in fs-repo but the rest in index-db. I have to use an FS path like /home/stephen/ipfs/ to make fs-repo work, but this means the name in index-db will also be /home/stephen/ipfs/. You could imagine other combinations of stores where choosing a common repo path would be impossible (e.g. required vs disallowed chars).

Wouldn't it make more sense to either a) allow each store to specify its own key, or b) just use one store for the whole repo?

[daviddias]

I made it this way because go-ipfs uses flatfs and levelDB simultaneously, which invalidates option b). On option a), fine by me, it didn't hurt anyone so far, but if you have a reason to add that to the API, I'm happy to take that PR :)

[hackergrrl]

"repo root" has no meaning in js-ipfs then, correct? The repo here is defined as a map of abstract-blob-stores. Since there is no common store there can be no "root". It sounds like "repo init" is just "make sure that exists() fails on all stores".

[hackergrrl]

Another question!

Why should we exposed ipfs repo init? Is there any reason why we'd want users to use this? Doesn't ipfs init cover this?

[daviddias]

re: ipfs repo init yeah, we don't really need to expose that one :) (It was added to the list of features on the API spec some time ago)

[daviddias]

It would make sense to upgrade https://github.com/ipfs/js-ipfs-repo to be able to tell stats about the repo.

repo gc still requires the pinning still :)