Support ipfs-api again

[haadcode]

For various reasons, we dropped the support for ipfs-api some months ago. We should bring it back.

One of the main reason to drop the support was the API mismatch between ipfs and ipfs-api especially when it comes to testing. The nodes are created in a different way and so this would require to re-factor the tests (mainly the before() parts where we start the IPFS nodes). There may be other dragons! 🐲 But from functional/technical perspective, it should work out of the box.

[nmarley]

Wow, had no idea this wasn't supported any longer, but got bit by this also today. @haadcode What help do you need getting this to work again?

[nmarley]

So it seems that with the write-permissioned database support, OrbitDB is looking like an entirely new product.

We've got a network of 4500 nodes and need to share IPFS peer ids in order to bootstrap the network connections. I had the idea to use OrbitDB in order to share those, but now with the addition of OrbitDB addresses, that's an additional layer which would have to be shared -- which is the point of us using OrbitDB in the first place, to avoid having to share unique values (the addresses) and add extra bootstrapping to our system. It's also now creating an "orbitdb" data directory, which was previously handled via ipfs (thru the ipfs-api).

Is it possible to restore these features from the previous version(s), or will this be the future of OrbitDB? I apologize for posting this here, but not sure where else to add design-related issues.

[haadcode]

Thank you @nmarley for posting and raising your concerns. This is absolutely the right place to do that. And thank you for bringing this up, I had no idea you were using or planning to use OrbitDB at that scale.

It sounds to me this was an important part of your plan, so if you want to resolve it as fast as possible, please reach out to me on IRC or email me at haad@ipfs.io and I'd be more than happy to help.

We definitely would like to support your use case, so let's try break it down in order to figure out what's possible and what's needed:

I had the idea to use OrbitDB in order to share those, but now with the addition of OrbitDB addresses, that's an additional layer which would have to be shared -- which is the point of us using OrbitDB in the first place, to avoid having to share unique values (the addresses)

Would you be able to elaborate on the mechanic of sharing the addresses? I don't understand why the OrbitDB addresses add an additional layer? If the addresses you intent to share are IPFS peer IDs, isn't that one database (thus one OrbitDB address)?

It's also now creating an "orbitdb" data directory, which was previously handled via ipfs (thru the ipfs-api).

This has always been the case: there's an orbitdb data directory and by default it's at ./orbitdb. However, you can change this by giving the constructor a path to the desired data directory: new OrbitDB(ipfs, './directory/to/db'). Does that help in your case?

Is it possible to restore these features from the previous version(s), or will this be the future of OrbitDB?

Would you be able to elaborate which features exactly are the ones that broke for you and you'd like to be restored? Based on your comment, I don't understand which features you mean and more importantly how are you using them. Perhaps there's a way to do what you want to do with the newer releases.

[nmarley]

Thanks so much @haadcode, I really appreciate the assistance.

Would you be able to elaborate on the mechanic of sharing the addresses? I don't understand why the OrbitDB addresses add an additional layer? If the addresses you intent to share are IPFS peer IDs, isn't that one database (thus one OrbitDB address)?

Sure. So we have a network of 4500 nodes, and want to ensure they can all connect/form their own network via IPFS. Every node already has the IPv4 address list for all the others, but we've also got to share IPFS Peer IDs (AFAIK). IPFS/OrbitDB would be great for this, but it's a classic chicken/egg problem. In order to get around this, I proposed a compromise - serve a basic API route which will return the IPFS peer id, but only query around ~10 of them upon initial bootstrap, enough to get a small network started.

The node list is (semi-)deterministic and each node has a unique ID, so it's possible to sort the nodes in such a way that each node connects to the 10 nodes with most similar unique IDs. Then, we can use OrbitDB to share the full list via KV store, something like:

// these values are unique per node
let hostname = 'node1'
let ipfs_peer_id = 'QmQQVKMgLUne717wVzkv4zWYQmr5GALv2FDGNzuQ11XCdf'

// add/update global node list/IPFS mapping
const kv = db.kvstore(namespace)
kv.set(hostname, ipfs_peer_id)

It's also now creating an "orbitdb" data directory, which was previously handled via ipfs (thru the ipfs-api).

This has always been the case: there's an orbitdb data directory and by default it's at ./orbitdb. However, you can change this by giving the constructor a path to the desired data directory: new OrbitDB(ipfs, './directory/to/db'). Does that help in your case?

Wow, I had no idea. I always ran our orbitdb PoC cases inside disposeable docker containers, so never dug into this. But you're right! Just tested our old PoC code from this summer (using orbit-db v0.17.3) and it's indeed creating an orbit-db directory for the kvstore namespace we used. And thanks for the tip! It will be nice to place this in a different data directory along w/our others.

Would you be able to elaborate which features exactly are the ones that broke for you and you'd like to be restored? Based on your comment, I don't understand which features you mean and more importantly how are you using them. Perhaps there's a way to do what you want to do with the newer releases.

So, right now it's probably just the ipfs-api interaction. Our system has several components, and the IPFS is actually shared between them, so it makes the most sense (and seems like the best architecture) to run a separate IPFS daemon and just connect to the API from both of the other components.

But also, with OrbitDB addresses, does that mean that our nodes need to share both IPFS Peer IDs and the OrbitDB addresses? I guess that's the extra layer I'm talking about. If we're just able to use a single namespace without worrying about sharing the Orbitdb address, then it makes it much simpler to just query the kvstore based on that namespace.

I should note: we have developed a separate process to wrap the payload and sign it cryptographically, to ensure the payload isn't spoofed by a malicious actor.

Thanks so much for the help, and if there's anything we can do to help WRT making the ipfs-api work again, would love to assist.

[haadcode]

@nmarley thanks for the details! I'll take a look at what it takes to make it work again with ipfs-api and report back and follow up on more questions re. how you plan to do it. How urgent is this for you?

Quickly though: what's the namespace in your example above? Is that known beforehand, is it static and how is it formed? Are all those 4.5k nodes in that namespace? Can anyone read/write in that namespace or are the writers/readers predefined and known or dynamic? Who signs the payloads you mention? Do the nodeIDs/hostnames/ipfsIDs change or are they fairly static? I'm trying to understand better how authentication happens in your model and what is connected with what in terms of keys :)

[nmarley]

thanks for the details! I'll take a look at what it takes to make it work again with ipfs-api and report back and follow up on more questions re. how you plan to do it. How urgent is this for you?

Thank you! And it doesn't have to be urgent, meaning we could theoretically build a network using a basic express API and some rules for connecting to specfic nodes based on the (semi) deterministic node list. But OrbitDB would make this a bit easier I think.

Quickly though: what's the namespace in your example above?

In my example it's a static value such as tld.orgname.product.ipfs_kv1

Are all those 4.5k nodes in that namespace?

Yes, that's the plan, at least for now

Can anyone read/write in that namespace or are the writers/readers predefined and known or dynamic?

Original plan (this is before permissioned DBs were a thing) is that anyone can read/write, but we verify the cryptographic signatures. We have the pubkeys, IPv4 and unique ID for each node in the list.

Who signs the payloads you mention?

We've devised a wrapper where each node signs a payload and the resulting message added looks something like this:

{
  "unique_id": "b897f67af94054a9b71765bb8f9f0bef641c149db30719053637ef1178378d4a",
  "payload": "base64-encoded here...",
  "sig": "also base64-encoded here"
}

Do the nodeIDs/hostnames/ipfsIDs change or are they fairly static?

The Node IDs should be fairly static, as well as the IPFS Peer Ids.

I hope this makes sense, please let me know if something is still unclear.

[haadcode]

@nmarley I'll reach out to you in PM to chat a little more about your system to understand it better.

Meanwhile, I had some time yesterday to look into this. As far as I can tell, there's two tasks we need in order to get OrbitDB working with ipfs-api again:

• Run tests with ipfs-api (using js-ipfsd-ctl to spawn go-ipfs daemons). This is essentially re-factoring the tests so that they run with both, js-ipfs-api and js-ipfs. This in practice means re-working the before/after parts in every test. Shouldn't be too complicated but does require a bit of work. But we won't be able to get the replication tests running without:
• Implement direct-connection mechanism that we get from ipfs-pubsub-room in ipfs-api compatible way (More details: Make ipfs-pubsub-room work with js-ipfs-api ipfs-shipyard/ipfs-pubsub-room#28, tl;dr we use libp2p primitives in pubsub-room that are not exposed by ipfs-api). To make this work with ipfs-api, it means essentially writing a new component and replace the direct-connection part of ipfs-pubsub-room. This is trickier as it touches bunch of layers and modules and comes with trade-offs and design considerations explained in the issue.

I'll give this (mainly the second task) a bit more thought and if it still feels like the best approach to solve this in short-term, I'll go ahead and write the component we need (along with changes needed in other modules, if any).

If anyone wants to help, getting the test running with ipfs-api/ipfsd-ctl would be hugely helpful! Note that none of the replication tests will run without the new component for pubsub-room, but at least we can test the rest.

[haadcode]

Got some good news: a PR landed in master which contains everything to support go-ipfs (js-ipfs-api) again!

I'll hold off making a release for a few days and keep testing it. Would love to get some more eyes on it and report back any problems you may encounter.

The next release (0.20.0) will contain the added support.

[fazo96]

Hey @haadcode, was playing around with using go-ipfs through ipfs-api instead of js-ipfs, running with orbit-db master branch.

However I ran into this: ipfs-shipyard/ipfs-pubsub-1on1#1 which blocks orbit-db from working in this environment

[haadcode]

A small update on this: we have the support for go-ipfs again starting 0.19.9 but as we're hitting a bug with the tests (#417), can't guarantee that it'll work.

In short: you should be able to use orbit-db again with go-ipfs, but YMMV.

We'll make the support "official" once the tests are passing CI confidently.

[haadcode]

This hasn't been updated in a while, but we're still blocked by a bug in go-ipfs (libp2p/go-libp2p-pubsub#93), so can't finish #417 reliably.

The good news is that everything seems to work.

What's blocking us to officially support it is that due to the aforementioned bug, some of the tests (replication tests) fail about 50% of the time, so we can't be sure that everything checks (100%).

That said, some people have been using orbit-db for a while now and it seems to work, so hopefully that doesn't prevent orbitdb users to try it out! 👍

[telackey]

I am attempting this, but am unsure if I am encountering a bug or misunderstanding/misusing the intended functionality.

The ultimate goal is that I have a nodejs microservice in Kubernetes, which I would like to switch to use our IPFS nodes for key/value storage. These Kubernetes pods with the microservice are fairly stable, but still need to support reloading their databases from scratch when redeployed, while the IPFS nodes are up 24/7.

As a test, I set up a local go-ipfs daemon, created an ipfs-api client instance configured to talk to it, which is passed to the OrbitDB constructor. All appears well at first, and I can successfully get and set key/value pairs.

There is a good bit of traffic flying around on port 5001, mostly to /api/v0/object/get and /api/v0/object/put, and a bit for pubsub.

However, if I delete the orbitdb directory in the cwd, the next time I attempt to open the database with orbitdb.open('/orbitdb/Qm...');, while it completes successfully, the database is invariably empty after db.load, which is certainly not expected.

EDIT

On closer inspection, and after reading a good bit of the code, it seems my issue is more closely related to this: https://github.com/orbitdb/orbit-db/issues/352

[aphelionz]

I believe this can be closed as we support js-ipfs-http-client now