Call for help - Limitations with js-ipfs

[satazor]

Hello everyone!

This is a call for help 🙏. As some of you know, I'm going to give a workshop about IDM and Nomios in IPFS Camp. In a glimpse, IDM is a concept and specification for an Identity Wallet, based on DIDs, Verifiable Credentials and DID-Auth. Nomios is a Identity Wallet based on IDM, but anyone can build their own.

One of Nomios' goals is to work within the browser, without requiring users to install or setup external tools, such as IPFS Companion or JS/GO daemons. This doesn't mean that Nomios will be usable solely in the browser. In fact, the closer to the OS the better as it gives us a better user-experience, more capabilities and increased performance. Though, asking users to install something "native" - like an Electron based app - will yield better results after they are "sold" into the product.

This is where I need help. Here's a list of things I use from IPFS that must work within the browser:

The js-ipfs route

ipfs.dag.get & ipfs.dag.put

We are using OrbitDB to store and replicate data amongst wallets, which in turn uses ipfs-log. That module uses the DAG API to build a immutable log based on a Merkle DAG. Moreover, the DAG API is also being used by did-ipid to store DID Documents.

❌ Using the latest version of IPFS in the browser, I'm unable to do ipfs.dag.put in one browser and fetch it using ipfs.dag.get in a different browser. The promise takes forever to resolve and eventually times out. The behavior is the same when attempting to resolve from the ipfs.io public gateway. My understanding tells me that this should work without any tinkering since preload is enabled by default, making the content available to bootstrap nodes faster. Here's a codepen for everyone to try: simply open that URL in Chrome and Firefox and try to resolve the CIDs from each other.

✅ You will notice there's some commented code in index.js file of the codepen, which defines a Swarm value pointing to a Websocket Star server. This makes things work but isn't this a bit centralized since all discovery and traffic goes into it? I'm ok in using this for the workshop, but I'm interested in hearing alternatives. Shouldn't preload make things work "magically"? Shouldn't WebRTC help here?

ipfs.add & ipfs.get

The FILES API is used to store blobs, such as the identities images.

❌ The problems exhibited here are the same as ipfs.dag.get & ipfs.dag.put.

✅ By using a custom Swarm pointing to a Websocket Star server, fetching files are pretty fast.

ipfs.keys, ipfs.name.publish and ipfs.name.resolve

One of the first DID methods we are supporting is ipid.

(ipid) is and implementation of the DID (decentralized identifiers) specification over the IPFS (Interplanetary File System) network using the IPNS (Interplanetary Name Service) cryptographic namespace resolution service.

IPID based DIDs have the following shape: did:ipid:<ipns-name>. To resolve these DIDs to DID Documents, we need to resolve the IPNS record. Moreover, to both create or update a DID Document, we need to publish the IPNS record to point to a DID Document's CID.

The IPNS key is not the IPFS node identity. Therefore, we need to be able to temporarily import the key using ipfs.keys to publish the IPNS record. After the publish succeeds, the key is removed. This makes the republishing impossible but this will later be solved with improvements to IPNS that allow for third-party republishing. For now, we are setting a high lifetime (1 year) to circumvent this limit, but we don't know if it's being respected. ⚠️Can someone clarify this point?

❌ Publishing of IPNS records succeed but then I'm unable to resolve them in a different browser, failing immediately saying that it's unable to resolve the record. The issue I think is that we don't have DHT enabled yet and it's blocked until #1984 is fixed.

✅ I'm inclined to use the work that @hugomrdias did on a tiered IPNS approach where he used CloudFlare workers (or wait for the DNS based approach). I haven't yet fully tested it in the browser but it seems to work within the JS daemon. This is indeed a bit centralized but it works reliably if we have an internet connection.

PubSub

PubSub is used by OrbitDB to replicate data between wallets.

⚠️✅ PubSub has been working nicely but it requires currently setting (I think) a custom Swarm value pointing to a WebSocket Star server. This is a similar requirement than the one mentioned in ipfs.dag.get & ipfs.dag.put, but I would prefer peers to connect directly via WebRTC or similar than using a WebSocket Star to relay traffic.

---

The external daemon / go-ipfs route

Besides the solutions mentioned above, I tried to explore other ways, including ones that go against the original goal of not requiring setting up or installing external tools:

• IPFS Companion with an external daemon

❌ ipfs.name.publish and ipfs.keys are not yet supported, see ipfs/ipfs-companion#452.

• IPFS HTTP Client with an external daemon

❌ Doesn't support PubSub in browsers yet, see: ipfs-inactive/js-ipfs-http-client#518.

---

This was a summary of my investigation. Please keep in mind that this thread is not to blame anyone but to improve the project for use-cases such as mine. I count with everyone's experience to find solutions other than the ones I mentioned. 🙏

//cc @alanshaw @daviddias @pgte @vasco-santos @hugomrdias @jsoares

Feel free to mention anyone relevant to this discussion.

[satazor]

//cc @lidel

[daviddias]

@satazor the underlying cause to all the confusion (and probably a ton of debugging hours wasted) is that you have an expectation that a js-ipfs node without any extra configuration, will be able to connect to any node in the network, magically.

This is completed fair and granted and it is the world we want to be, however, challenges with implementing a DHT that doesn't make the browser explode, makes things complicated.

When you set up something such as a *-star transport, what you are adopting, is a rendezvous point (think of it as the local bar) that you and all your friends go to meet with each other and that is why once you set up that transport, you immediately start seeing communication flowing.

Today what you have is:

What you want (everyone wants) is to have the DHT and Relays configured from the start

Now a few questions for the @ipfs/wg-js-core:

• Will DHT be enabled soon?
• If not, why not have delegated routing enabled?
• Do we have tests for the both above?

[daviddias]

Also related:

• Cannot retrieve content via the DHT #1984
• Getting content from browser node to gateway using 0.35 RC #1923

[jacobheun]

DHT

The DHT will be enabled soon, however it is still going to be somewhat slow for browsers and nodes behind NATs. Until we can get improved support for direct dialing other browsers, like hopefully 🤞 via WebRTC next arewedistributedyet/arewedistributedyet#22 (comment), queries on the network take time.

The peer discovery component (randomWalk) that we'll get with DHT will help improve the breadth of our nodes networks. We have improved connection management in libp2p, so turning this on in the browser should be reasonable.

I know @kumavis has been playing around with DHT queries and webrtc, so he might be able to add his experiences there.

Another thing we will likely need to help improve this is Connection Manager tagging in js-libp2p. As the DHT will result in us discovery a lot of peers, there's a risk there that the Connection Manager will terminate connections to nodes that we really want to keep. Being able to tag a node as Critical or something similar, would enable us to try and keep those connections as much as possible.

Gateway/Preload Performance

There is also a possibility that the gateway/preload nodes are overloaded with connections. If they have too many connections and keep culling connections on their end, it's going to be difficult for us to pull content down.

Delegated Routing

If not, why not have delegated routing enabled?

This can definitely be done, provided we have the nodes running with support for this. js-ipfs previously didn't have refs support. I believe it was recently added, I'm not sure if it was added to the api yet though. If so, we could test the libp2p example against a local instance to verify.

[daviddias]

@jacobheun I need a clarification. Why is refs on js-ipfs important for the Delegated Routing? I believe that the intention is for the "Delegated Nodes" to be go-ipfs ones and not js-ipfs. js-ipfs would use those nodes to do Delegated Peer Routing and pre-fetching.

Do we have tests for the delegated routing functionality? That test with the real public infrastructure?

[alanshaw]

My original plan was to release JS IPFS 0.37 with the DHT in time for IPFS Camp.

Content preloading should hopefully be working better by the time camp comes round. The preload nodes are currently running an old (possibly broken) version of go-ipfs. I've been talking to infra and when 0.4.21 is released they'll be updated at the same time. This will happen for future releases too.

Are there known relay nodes deployed that we can connect to or do they need to be discovered via the DHT? It's not great if we have to discover a relay before nodes behind NATs can talk - there's no telling when (or if) it'll happen.

[alanshaw]

@jacobheun we also need connection tagging for the preload nodes - currently they are part of the bootstrap nodes but connection manager could easily prune them when DHT comes along.

[pgte]

Re. landing the DHT, I'm proposing to help on #1994

[satazor]

@daviddias I will be answering inline

@satazor the underlying cause to all the confusion (and probably a ton of debugging hours wasted) is that you have an expectation that a js-ipfs node without any extra configuration, will be able to connect to any node in the network, magically.

I think we could improve the DX in these situations. Developers evaluating IPFS in the browser will experience the same issues and most of them will give up. While things don't work automatically, perhaps we could print a console.warn saying:

"It seems you are running JS IPFS within the browser without a suitable discovery & transport, meaning other peers won't be able to see you nor send/receive data. Read more about this in: <Link>" - where <Link> contains more information about the subject.

Today what you have is:

In the diagram you drafted (thanks for the effort btw), the IPFS nodes within the browser are connected to the bootstrap nodes & gateway. Shouldn't I be able to pull content I'm pinning from the gateway? I've tried it and it's not working (the request is pending forever).

What you want (everyone wants) is to have the DHT and Relays configured from the start

Understood 👍. What's the stability of the WebSocket Star servers? Can I use these or should we deploy new ones specifically for the workshop?

    // ws-star0.ams = "ws-star0.ams.dwebops.pub ws-star.discovery.libp2p.io"
    // ws-star1.par = "ws-star1.par.dwebops.pub"
    // ws-star2.sjc = "ws-star2.sjc.dwebops.pub"

---

As @jacobheun said, I think focusing on WebRTC next is a great idea for the browsers use-case. Realistically speaking, do you think we have time to solve the issues we are having with the DHT and do a release with it before IPFS Camp?

[jacobheun]

Why is refs on js-ipfs important for the Delegated Routing?

Having refs on js isn't important, but it would be nice to have. Yes, the original intention was to expose those on go. The support is there, but it still hasn't been deployed, afaik.

The example in libp2p can and should be pointing at the public network, however the gateway doesn't have the configuration to turn the needed endpoints on at the moment, the needed endpoints 404. The api also appears to not be hanging. The curl mentioned at https://github.com/ipfs/go-ipfs/blob/master/docs/gateway.md#read-only-api, just hangs.

@jacobheun we also need connection tagging for the preload nodes - currently they are part of the bootstrap nodes but connection manager could easily prune them when DHT comes along.

The thought is to be able to configure the list of critical peers for you, so we could just do that in the js-ipfs config. Ideally we wouldn't need this, but it's a stop gap to get us to the right place.

[jacobheun]

As @jacobheun said, I think focusing on WebRTC next is a great idea for the browsers use-case. Realistically speaking, do you think we have time to solve the issues we are having with the DHT and do a release with it before IPFS Camp?

There are improvements we can get out that will help, but realistically the DHT is going to be slow until we can get direct connections to peers quickly. In small networks the DHT would be relatively fast, because the paths are short, but if we're in the global network it's going to be slow. We also have to contend with dead/undialable nodes in the DHT that can block.

If the network being demo'd on is isolated, we should be able to get reasonable response times.

I put together a simulation as part of the recent DHT updates we did, that mimic the public network as we understand it today. Even with the improvements you're potentially looking at upwards of 1 minute query times. I received similar response times executing queries on the public network.

[jacobheun]

Understood 👍. What's the stability of the WebSocket Star servers? Can I use these or should we deploy new ones specifically for the workshop?

@satazor they should be fairly stable. I'm currently working with infra to take hand off of those. We might move those to new machines and give better dns entries to the latter two, but they should be fine for demoing on. If we want to isolate the network to help better control query times, then servers specifically for the workshop might help.

[dirkmc]

It's also worth mentioning that DHT performance depends on the go implementation being performant, and maintainers updating their local instances, as there are currently thousands of live go DHT nodes behind NATs that time out, clogging up DHT queries. eg see: libp2p/go-libp2p-kad-dht#88 ipfs/kubo#3860

[daviddias]

Similar to http://github.com/ipfs/interop, I believe it is time to have a http://github.com/ipfs/integration for integration tests on pre-release, that test new versions of IPFS (both JS and Go) against the Live Network.

[satazor]

An update regarding IPNS: I've successfully tested https://github.com/ipfs/js-ipfs/tree/feat/ipns-over-workers branch on the browser. It's really fast because it contacts a simple REST API, but I think it's a good compromise for the workshop even though is a centralized approach.

[jacobheun]

Regarding webrtc, there is a spec proposal for having any libp2p node support signaling, libp2p/specs#159. This could potentially help a lot with connectivity of the network and would rely less on central servers. We could still use bootstrap nodes for the initial network discovery, but if a significant portion of the network supported the protocol that could be really impactful. This could be a decent stop gap until/if we get some level of listening support in browsers.

[alanshaw]

npm i ipfs@camp with websocket-star enabled pointing at a rendezvous server for camp?

[satazor]

@alanshaw the IDM wallet is currently configured with one of the public websocket star servers. Ideally, it would be a local machine running on the camp to circumvent any internet issues, but the problem is IPNS. @hugomrdias’ CloudFlare IPNS is working but requires internet connection. IPNS is used to resolve DID documents in order to verify signatures.

[daviddias]

As note to readers, more context on the current infrastructure design can be found at #1459

[jhiesey]

Some use case proposals here: libp2p/js-libp2p-webrtc-star#177 (comment)

[SgtPooki]

js-ipfs is being deprecated in favor of Helia. You can learn more about this deprecation and read the migration guide.

Please feel to reopen with any comments by 2023-06-02. We will do a final pass on reopened issues afterwards (see #4336).

This is a great issue that should be reviewed at least one more time before we completely deprecate js-ipfs @BigLep @achingbrain. Any learnings we need to take on to Helia that haven't already been done?