[EPIC] IPNS and Package Managers

[aschmahmann]

In order for package managers to make their content available to users the registry information about a given package needs to support:

1. Discovery: Users know where to find a package (e.g. go-ipfs as published by the IPFS team)
2. Updates: Users know where to look for the latest package (e.g. I have go-ipfs 0.4.1 and want to see if there's a version later than 0.4.1)

Discovery can be used to replace Updates entirely, by just insisting that users do whatever they did for discovery any time they want an update. However, discovery tends to be a much harder and more user intensive process than updates (e.g. a Github search for go-ipfs and choosing the correct fork, as opposed to going to github.com/ipfs/go-ipfs/releases) and so it is helpful to separate these two concerns. For the time being we are going to punt on Discovery and focus on Updates.

IPNS is designed for updates and so is a natural fit for Updates in IPFS based package managers. Some of the IPNS issues we should tackle to properly support package mangers are:

1. Performance via non-DHT Transports:
   • PubSub as an independent IPNS transport (IPNS over PubSub as an independent transport ipfs/kubo#6447, not yet started in JS)
   • IPNS over DNS (IPNS over DNS ipfs/js-ipfs#2000, go underway @djdv )
2. Features:
   • Pinning IPNS records (so that we restart following them on daemon reload)
   • Pinning IPFS objects referenced by IPNS (e.g. IPNS pinning: ipfs name follow ipfs/kubo#4435 and related referenced issues)
   • Third party republishing (Ability to republish third-party /ipns/ records ipfs/kubo#3117) (Solved within the PubSub transport)

There are more IPNS issues to tackle, but most of these really need to get off the ground for it to be relatively painless to run a package manager using IPFS/IPNS.

[djdv]

IPNS over DNS (ipfs/js-ipfs#2000, go underway @djdv )

I have a related repo around this here: https://github.com/ipfs/go-libp2p-dns-router
It resolves content records via subdomain cid's over DNS, but is in need of a spec to make sure implementations are aligned.
(There's some oddities in that current implementation)

e.g.
resolves bafybeigv6xgwkfhx3abfsuayuicb3l7xblpzrsdvjlt33oap43joacglyu.dns.ipns.dev => /ipfs/QmZFLGKTiYvxhCAQGDDSCvRPYeQCTaQMBAPL1Cqb8S695p

[meiqimichelle]

This is the one that's happening now: PubSub as an independent IPNS transport (ipfs/go-ipfs#6447, not yet started in JS) -- no blockers at the moment, just trucking along.

[aschmahmann]

As part of ipfs/kubo#6447 we landed libp2p/go-libp2p-pubsub#190 🚢 🚀

[aschmahmann]

Related to ipfs/kubo#6447 I will be putting together a spec for IPNS-over-PubSub as an independent transport now that it's pretty close to stable.

[meiqimichelle]

Update: we've asked @hugomrdias to move writing specs for ipfs/js-ipfs#2000 up on his priority list so we can move forward with IPNS over DNS.

[meiqimichelle]

@aschmahmann re: your question about which of the feature work to start on, once you're in a good place with the performance work --

You've already got these on your list:

• Pinning IPNS records (so that we restart following them on daemon reload)
• Pinning IPFS objects referenced by IPNS (e.g. IPNS pinning: ipfs name follow ipfs/kubo#4435 and related referenced issues)
• Third party republishing (Ability to republish third-party /ipns/ records ipfs/kubo#3117) (Solved within the PubSub transport)

And you reference:
There are more IPNS issues to tackle, but most of these really need to get off the ground for it to be relatively painless to run a package manager using IPFS/IPNS.

I'm chatting with @andrew -- we'd like to know more about what those other things are on your mind. Also, we are interested in what it would take to be able to track the history of changes connected to one IPNS record. What we're getting at -- when someone starts using IPNS, what's their user experience like? Can we articulate descriptions of what user stories will look like when folks start using it? Then, we can research them, and understand which are relevant to package managers. Some related musings happened on Cluster re: storing certain metadata along with objects, and Qri's storing history through storing the transform on the data (so it's reproducible and trackable).

[aschmahmann]

what it would take to be able to track the history of changes connected to one IPNS record

Good news, you can sort of do that already. IPNS is basically (key, value, sequence number), however the value can be anything you want. If the owner of the IPNS record (there must be an owner because we use sequence numbers to indicate versioning) decides that each new value will be an IPLD object that points at the previous data. Take the following example:

1. Create IPLD object {DataPath : /ipfs/bafyFirstFile, Previous: nil}
2. Create IPNS entry {Value : /ipfs/bafyA, SeqNo: 0}, where bafyA is the CID of the above object
3. Create IPLD object {DataPath : /ipfs/bafySecondFile, Previous: /ipfs/bafyA}
4. Create IPNS entry {Value : /ipfs/bafyB, SeqNo: 1}, where bafyB is the CID of the above object

Unfortunately, there's no way to prove that the IPNS publisher hasn't lied to you about the history (outside of already having a conflict copy of the history). This isn't generally too bad because IPNS is set up to only be able to properly deal with a single writer and so that writer is assumed to be trusted to manage their own data stream.

Some other things that would be great to tackle:

1. Operational Transform / CmRDTs (i.e. signed multi-writer collaborative data structures with history)
   • A lot of this work is already done at https://github.com/aschmahmann/ipshare, but it still needs some time for a few things including
     • More efficiently use the pubsub PRs that have landed
     • Clean up the code and try to make the interfaces a little friendlier
     • Would be really nice if I could use go-graphsync here instead of the simplified one already in use
   • This can be used for all sorts of things from real-time collaborative applications (e.g. PeerPad), to history tracking applications like Git
2. CvRDTs and/or delta CRDTs (i.e. unsigned efficient collaborative data structures)
   • peer-base and PeerPad use this framework and it would be nice to have something similar in Go, and generally available using supported libp2p contstructs
3. CmRDTs have problems relating to the history growing over time (like cloning an ancient highly active Git repo). Figuring out some reusable strategies for when/how to squash all the changes together would be great.
4. ACLs: Access control is very difficult to define and work with in a distributed system, however, there are likely a number of less daunting tasks we can handle

There's also the issue of specs, the IPNS specs need a bunch of work:

1. IPNS with multiple resolvers ipfs/specs#198, defining how multiple IPNS resolvers play nicely together (DHT, DNS, PubSub, etc.)
2. Improve IPNS spec ipfs/specs#205, what does IPNS actually mean?
3. IPNS Key Revocation ipfs/specs#219, what happens when crypto keys go bad?

[aschmahmann]

Started PRs for two specs related to IPNS-over-PubSub:

1. IPNS-over-PubSub as an Independent Transport ipfs/specs#218
2. [Working Draft] Fetch protocol spec libp2p/specs#204

[aschmahmann]

Update: As part of ipfs/kubo#6447 we landed libp2p/go-libp2p-pubsub-router#33 🚢 🚀

This update essentially gives us IPNS-over-PubSub as an Independent Transport. However, without making some modifications to go-libp2p-pubsub-router or landing libp2p/go-libp2p-pubsub#184 it'll be hard to see the effects of this on the public network because the code that uses the DHT to discover peers for the PubSub topic times out before the DHT query is likely to complete.

Therefore... next up is libp2p/go-libp2p-pubsub#184 (and the associated go-libp2p-discovery PR that will be associated with libp2p/go-libp2p#707)