Proposal: Reliable Mutability Primitive

[Gozala]

Originally draft was edited at https://hackmd.io/@gozala/ipns-pitch

@carsonfarmer provided some valuable feedback, which I'll migrate here for the visibility of others

cc @hugomrdias @aschmahmann @lidel

[aschmahmann]

This is a closely related issue ipfs/kubo#4435

[aschmahmann]

Note: Textile threads utilize pubsub which means that any issues with enabling pubsub by default are things that would be experienced by Textile threads users. They also utilize IPNS over PubSub for buckets.

[aschmahmann]

This is sort of doable already "for free" if we have some way to pin/follow IPNS updates. I can just share the private key with my multiple devices via ipfs key export/import.

There are alternative approaches too, but figured I'd point this one out.

[Gozala]

Yeah I meant getting it to a state where we can comfortably tell users this is how you go about it. I do have some reservations towards key sharing, but that's probably discussion worth having when we get there.

[aschmahmann]

At the very least because updating DNS is such a pain we see people using DNSLink and pointing it at IPNS records instead of IPFS because its so much easier to work with IPNS updates than DNS updates. Similarly, we may see people storing IPNS addresses in ENS to save on not wanting to publish many blockchain transactions.

[aschmahmann]

Note: likely includes an up-to-date IPNS over PubSub implementation

[Gozala]

@aschmahmann do you mean something like

Suggested change

	- Functioning IPNS in browsers
	- Functioning IPNS in browsers (over up-to-date PubSub implementation)

Or a separate item ? I

[aschmahmann]

Well IPNS over PubSub has a few tweaks over just PubSub itself to give some reliability. It's specced here.

@hugomrdias would probably know more about other work required to get IPNS functioning in a browser I just know that without the IPNS over PubSub changes in the spec that it won't be fast enough.

[BigLep]

Sorry for the basic question, but I'm still confused by what is meant by "functioning IPNS in browsers". What isn't working browsers today?

[Gozala]

I should probably capture this in the document. Currently pubsub is disabled in go-ipfs so all the IPNS records are published and resolved from DHT. Nodes in browsers do not have necessary capabilities to query/publish DHT. Instead they do publish and resolve IPNS records over pubsub. Result is fragmented network where go and js nodes see each others records.

Furthermore because because there is not enough nodes on the network with pubsub enabled published records from browser don't always make to nodes subscribed to them.

P.S.: Some of this may not be completely accurate, @hugomrdias @aschmahmann could you please confirm I'm not misrepresenting current state.

[aschmahmann]

Interesting, I'm not sure this is really required as we can give IPNS the same guarantees as IPFS. However, since the amount of data is smaller this idea of a public IPNS pinning service seems pretty cool.

[aschmahmann]

Also, more advanced mutability systems could be built (e.g. where instead of a single public key, we can have M of N signers required on records). But it all starts from having the UX and process around pinning mutable names and having a performant and efficient mutability system (e.g. Persistent PubSub, like what IPNS over PubSub uses)

[aschmahmann]

True, although just getting things stable + performant in go-ipfs and web browsers would likely go along way while collabs conversations are underway.

[BigLep]

@Gozala : can we put some high-level estimates (in terms of dev weeks) around each item in the "Brief plan of attack" above? I think that will help give a bit more clarity on the size of this project.

[Gozala]

I'm afraid I'm not well qualified to do this. I've mostly put this proposal together from prior discussions I had with @aschmahmann and @hugomrdias (still in outsider capacity) in an attempt to rally folks who have much better understanding of internal details.

Help trying to scope it out further, would be greatly appreciated.

P.S: Me & @hugomrdias plan no going over this proposal tomorrow to flush it out further.

[Gozala]

@carsonfarmer: This is still correct, IPNS is too unreliable for anything with essentially sub-minute temporal resolution. So for data updates, it is not useful. However, we are finding it quite useful for buckets, and have used it to great effect recently.

[carsonfarmer]

I should also mention something else to provide further context: We're currently managing 1000s (yes, that's right) of IPNS updates for buckets. This currently takes an extremely long time to republish. We are actively conducting tests to determine exactly how long this takes, and how often it fails to complete. In some cases, it is taking hours per day.

[Gozala]

@carsonfarmer: Yeh, this one shouldn't be ignored... maybe the existing solution is just not going to cut it in the end, and a fresh start is required. How likely is it that we could see "two" official methods, and have one of them win out?

[Gozala]

@carsonfarmer: Obviously I like this idea. But it would end up being something threads-like I think. A benefit here is that threads supports multi-writer updates, which I think its a feature lacking in IPNS right now. The downside is that threads "as is" won't scale. Additionally, the encryption layer in threads is too heavy handed right now for a general solution like this. Having said all that, these things are all fixable implementation details.

[Gozala]

@carsonfarmer: This would be an interesting shorter term solution

[BigLep]

(new person coming up to speed): what would this look like? I think I'll be able to grok it with a tiny sketch or a mock sample code example.

[Gozala]

We have recently created pinning service API that allows you to add pinning services like pinata to your IPFS node and then remote pin things. Kind of like adding git remote and pushing changes to it.

Idea here is to extend that API in a way that you could update DNSLink provided by the pinning service to the new CID through extended API. Kind of like when you push to remote it updates corresponding branch to new commit. In our case branch names would be DNSLinks, that you could update to point to new CID.

@BigLep does this answer your question ?

[BigLep]

@Gozala : thanks - this helps. The only open area for me then is where is this DNSLink stored? Is this a DNSLink in a DNS record on a (sub)domain that the Pinning Service owns?

Basically would the chain be
My domain (e.g., mywebsite.com) has DNS record that points to a domain the Pinning service controls (e.g., mywebsite.somepinningservice.com) which has the DNSLink to the latest CID?

[lidel]

@BigLep This is great question – it makes us think about all implications.
Depends on: if you (A) publish to your own domain, or (B) use a subdomain provided by pinning service for free.

To give you a real life example, let's say we want to host https://cid.ipfs.io on some pinning service (pin + take care of updating dnslink + TLS).

(B) is easy. Pinning service simply give you a name on some domain they own. Fleek does that already, they give https://cid-utils-website.on.fleek.co/ for free + they set up DNSLink for those names (because they own the top domain).

(A) means we want to delegate updates of DNS TXT record on _dnslink.cid.ipfs.io to some pinning service.
This can be done either by:

• giving DNS provider credentials to pinning service + expecting them to implement support for proprietary APIs of each dns provider (not realistic IMO, but mentioning here for completeness)
• using DNS protocol itself for delegation (details TBD, below are broad stroke ideas to explore):
  • setting ALIAS record, delegating everything to a subdomain controlled by pinning service
  • setting NS records that delegate management of _dnslink.cid.ipfs.io to a DNS server controlled by pinning service. That way they have full control over DNSLink without having any control over other records (because we delegated only _dnslink subdomain)

[BigLep]

Thanks a lot for putting this together - good stuff. Apologies if the questions below are all basic. Feel free to point me anywhere I should be reading to self-service answer.

[BigLep]

(New person comment): is this essential for unblock the core use-case of enabling customers to do ipns updates instead of DNS updates?

[Gozala]

Yes if those updates were to made from browser. It is also however useful on it's own merit as it enables apps to pull updated content even if such content has no human readable name.

[BigLep]

How do we get more data on this? Is this a question to PM's?

[Gozala]

I think it's well know fact, evidence been that teams have tried and ultimately end up going with DNSLink solution. I'm pretty sure PL did that too, but someone with longer history here might know better.

We could also get specific collabs to comment here to either validate the claim or dispute it. I have invited folks from textile & fission to validate. We could reach out to more teams.

@olizilla do you by a chance know if we used to have IPNS in DNSLink past or am I making wrong assumptions here.

[lidel]

@BigLep good pulse check in this issue (2017–today): ipfs/kubo#3860

[BigLep]

Have we been presented any usecases where ipns (once it has a reliable mutability primitive) isn't sufficient?

[Gozala]

• IPNS lacks human readable/memorable naming which is a concern in most human facing interactions.
• I have in the past voiced concerns about the lack of revision history associated with it (although one could built such a system on top).
• There also had been discussions about ability to have multiple publishers to the same name.
• And then there are also questions about compromised or lost key recovery

I suspect that despite all this usable IPNS will address enough use cases to be worth it. That said we might find that a higher level primitive that address more use cases out of the box will provide a better market fit.

[BigLep]

Is this even possible? Doesn't it depend on each DNS' API for mutating actions?

[Gozala]

Hypothetically speaking we can start a standardization effort form a committee and get enough big players onboard to get it done. I imagine there might be some value for DNS providers to make their products more accessible.

[BigLep]

(new person coming up to speed): what would this look like? I think I'll be able to grok it with a tiny sketch or a mock sample code example.

[BigLep]

Sorry for the basic question, but I'm still confused by what is meant by "functioning IPNS in browsers". What isn't working browsers today?

[BigLep]

@Gozala : can we put some high-level estimates (in terms of dev weeks) around each item in the "Brief plan of attack" above? I think that will help give a bit more clarity on the size of this project.

[BigLep]

Sorry to be slow, but why is it required to get pinning services onboard? I'm not saying we should skip doing this, I just want to make sure I understand the inter-dependency. For example, in my head right now I understand that Pinning Services are critical for persistence but I'm less clear on their role in mutability.

[Gozala]

That is a valid point and calling out pinning services specifically might be projecting too far of how things would work.

This was partially influenced by the fact that IPNS (at least in its current form) requires records to be republished otherwise they'll become unsolvable. Implicit assumption here, is that, unless there are motivated nodes to do this, system will end up impractical. Pinning services just seem well positions to do this given that they are motivated nodes keep content that IPNS will point to available (insert git branch analogy here).

That said we could and probably should consider different strategies to meet reliability goals.

[Gozala]

In other words I think we still need someone to help coordinate efforts beyond just leading implementation efforts, to ensure that we do not end up with a system that is directly tied to PL infrastructure. I am assuming this involves getting other teams onboard.

[lidel]

+1 on at least thinking about pinning services in the context of IPNS:

• how pinning of /ipns/{libp2p-key-cid} would work?
  (would there be special handling for a CID with libp2p-key codec etc)
• how they can help with avoiding IPNS rot
• how someone could pin IPNS record behind /ipns/en.wikipedia-on-ipfs.org (when we switch it to IPNS .. or not)

[BigLep]

Maybe add a note that this is part of the larger story in #42 ?

[Gozala]

I am not sure this would be accurate. I think there is a merit to reliable mutability primitive whether we decide to do human readable names or not.

[lidel]

I removed IPNS from #42 and would like to add some IPNS-specific things here (details below)

[mikeal]

i’d like a review approval from @aschmahmann on this one

[Gozala]

I think it is worth pointing out that this proposal was mostly about:

1. Documenting current state of mutability in IPFS
2. Making a case for addressing this problem

During iterations it became clear that different strategies (with various tradeoffs) could be used to address it. I have proposed #66 as one such strategy and was hoping that other proposals would surface as well. I also attempted to further document current limitations https://hackmd.io/@gozala/state-of-ipns across environments that I challenge other previously discussed strategies to improve IPNS across the board.

[mikeal]

@Gozala should we close this PR in favor of #66 then?

[Gozala]

@Gozala should we close this PR in favor of #66 then?

Not sure. I feel like current proposal structure does not support decoupling problem statement that attempts to clarify why we should address the problem, from concrete plan how can we do that. In many cases including this one it would make sense to have single problem statement (like this document) and multiple proposals to address it (which could link to this instead of reiterating).

Currently "Brief plan of attack" attempts to capture what @aschmahmann had been saying on various channels, to whom I'd defer this call.

[aschmahmann]

I feel like current proposal structure does not support decoupling problem statement that attempts to clarify why we should address the problem, from concrete plan how can we do that. In many cases including this one it would make sense to have single problem statement (like this document) and multiple proposals to address it (which could link to this instead of reiterating).

💯

IMO going down the path of this proposal better prepares us for a decentralized solution while also having "exit points" towards #66 if needed.

Briefly my understanding of this proposal is:

• Implement third party IPNS republishing
• go-ipfs default use of PubSub and IPNS over PubSub
• js-ipfs fully implementing IPNS over PubSub (i.e. spec which includes the fetch protocol) and using it by default

Briefly my understanding of #66 is:

• Define fast lane name resolution specification.
• Define name keeper service specification.
• Define name routing service specification.
• Implement name routing service in go-ipfs
• Implement name keeper service in go-ipfs
• Implement fast lane name resolution across web, go, node ipfs.
• Deploy name routing service to PL operated boostrap nodes

But what if instead these were almost the same:

• Define fast lane name resolution specification -> ✅ fetch protocol
• Define name keeper service specification -> third party IPNS republishing + "pinning API" to ask a third party to do it for you
• Define name routing service specification -> ✅ DHT provider records
• Implement name routing service in go-ipfs -> ✅ DHT provider records
• Implement name keeper service in go-ipfs -> third party IPNS republishing + "pinning API" to ask a third party to do it for you
• Implement fast lane name resolution across web, go, node ipfs -> IPNS over PubSub (Go ✅, JS ⚠)
• Deploy name routing service to PL operated boostrap nodes -> ✅ delegated routing

There's a reasonable point made in #66 that maybe IPNS records should have more information in them like routing hints (e.g. where is a good node to start my query), and while we can do this with pretty low effort because of how the DHT processes IPNS records I also don't think it's really going to be necessary once we've got IPNS over PubSub actually working.

There is a risk/issue with the IPNS over PubSub approach wherein PubSub falls over if you try and sign up for too many topics. However, if this becomes an issue we can easily pivot towards #66 by making a new router type that only uses the Fetch protocol instead of also using PubSub. The tradeoff is that we no longer get updates pushed to us and have to poll for updates, but it's an easy pivot to make if we need to.

[momack2]

Is this something we can grant to an ecosystem team? third party IPNS publishing sounds like something that could be executed through an RFP/devgrant...

…

On Wed, Mar 24, 2021 at 2:54 PM Adin Schmahmann ***@***.***> wrote: IMO going down the path of this proposal better prepares us for a decentralized solution while also having "exit points" towards #66 <#66> if needed. Briefly my understanding of this proposal is: - Implement third party IPNS republishing - go-ipfs default use of PubSub and IPNS over PubSub - js-ipfs fully implementing IPNS over PubSub (i.e. spec <https://github.com/ipfs/specs/blob/master/naming/pubsub.md> which includes the fetch protocol) and using it by default Briefly my understanding of #66 <#66> is: - Define fast lane name resolution specification. - Define name keeper service specification. - Define name routing service specification. - Implement name routing service in go-ipfs - Implement name keeper service in go-ipfs - Implement fast lane name resolution across web, go, node ipfs. - Deploy name routing service to PL operated boostrap nodes But what if instead these were almost the same: - Define fast lane name resolution specification -> ✅ fetch protocol - Define name keeper service specification -> third party IPNS republishing + "pinning API" to ask a third party to do it for you - Define name routing service specification -> ✅ DHT provider records - Implement name routing service in go-ipfs -> ✅ DHT provider records - Implement name keeper service in go-ipfs -> third party IPNS republishing + "pinning API" to ask a third party to do it for you - Implement fast lane name resolution across web, go, node ipfs -> IPNS over PubSub (Go ✅, JS ⚠) - Deploy name routing service to PL operated boostrap nodes -> ✅ delegated routing There's a reasonable point made in #66 <#66> that maybe IPNS records should have more information in them like routing hints (e.g. where is a good node to start my query), and while we can do this with pretty low effort because of how the DHT processes IPNS records I also don't think it's really going to be necessary once we've got IPNS over PubSub actually working. There is a risk/issue with the IPNS over PubSub approach wherein PubSub falls over if you try and sign up for too many topics. However, if this becomes an issue we can easily pivot towards #66 <#66> by making a new router type that only uses the Fetch protocol instead of also using PubSub. The tradeoff is that we no longer get updates pushed to us and have to poll for updates, but it's an easy pivot to make if we need to. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#19 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEXAF25J67CEFOWS4URVQLTFJNS5ANCNFSM4XXO2QTQ> .

[aschmahmann]

Is this something we can grant to an ecosystem team? third party IPNS
publishing sounds like something that could be executed through an
RFP/devgrant...

Maybe? The toughest part of the job is doing the plumbing in go-ipfs which shouldn't be too hard to do but is also not intuitive. I'd be ok giving this a shot as long as we do regular check-ins to check for alignment.

[Gozala]

But what if instead these were almost the same:

I think it is worth considering:

1. Complexity between approach proposed in propose: fast lane ipns #66 and below. Broadly speaking propose: fast lane ipns #66 attempts network querying by caching name keeper addresses forever at:

   1. Local level
   2. Name routing level

   And removes need for disturbing a network when publishing updates by making it point to point operation.

   Furthermore by embedding name keeper address in the DNSLink all human readable name resolutions could effectively be point to point operations.

   It is less decentralized, but but that is a conscious decision to remove all of the overhead in happy path. It does not replace need for fully distributed solution as it is still needed as a slower fallback.

2. Dependencies

#66 intentionally does not specify how name routing service API nor name keeper API nor how things get replicated. That is because I want to be able to make progress on IPNS without blocking on pubsub or DHT in JS or other things.

* Define fast lane name resolution specification -> ✅ fetch protocol

Not sure what you're referring to here. Assuming it's name keeper resolution I think fetch protocol is probably a good way to go about it. Although protocol seems generic enough that I feel some semantic meaning might be useful.

* Define name keeper service specification -> third party IPNS republishing + "pinning API" to ask a third party to do it for you

#66 intentionally makes publishing an operation that does not require disturbing the rest of the network. I think it is a good compromise, maybe it is not ? Worth comparing tradeoffs at least.

* Define name routing service specification -> ✅ DHT provider records

in #66 intention was to decouple name routing from DHT, pubsub etc... If node is aware of some name keeper(s) it can provide routing. It does not even needs to be IPFS node.

* Implement name routing service in go-ipfs -> ✅  DHT provider records

👍

* Implement name keeper service in go-ipfs -> third party IPNS republishing + "pinning API" to ask a third party to do it for you

In #66 intenion was that anything, as simple as cloudless lambda could play that role in the system. All it needs is a key value store as opposed to whole IPFS node.

* Implement fast lane name resolution across web, go, node ipfs -> IPNS over PubSub (Go ✅,  JS ⚠)

I really want us to arrive to a place where you can build distributed applications on the web without having to have to operate ipfs nodes on the server, because that is where many choose AWS as cheaper and simpler alternative. #66 was geared towards enabling that so might be worth considering.

It is true however that it creates a system in which a name resolution depends on an authoritative node, however remedy is that name owner can freely swap that authority which I think provides a good compromise.

* Deploy name routing service to PL operated boostrap nodes -> ✅ delegated routing

Load on routing nodes as per #66 is designed to be a lot smaller than I imagine it would be with delegated routing.

[BigLep]

Assigning to grants since it won't get worked on in the short term with the current program structure. @mikeal is going to evaluate grant possibility.