[NIP-47] Add versioning and migrate to NIP-44.

[jklein24]

NIP-04 is deprecated and no longer recommended for use. Authors of NIP-44 have confirmed that it can act as a drop-in replacement for NIP-04 for NWC's use case.

Versioning for NIP-47 allows the protocol to evolve and even make breaking changes while maintaining backwards-compatibility where needed. Versions are represented as <major>.<minor> (e.g. 1.3). Major version bumps imply breaking changes that are incompatible with the previous major version. Minor version bumps, however, only include non-breaking feature additions or improvements which can maintain full compatibility with the previous minor version.

[vitorpamplona]

Ack

FYI, NIP-46 (#1248) is just dynamically figuring out if the encryption is nip-04 or nip-44.

fun isNIP04(ciphertext: String): Boolean {
    val l = ciphertext.length
    if (l < 28) return false
    return ciphertext[l - 28] == '?' && 
           ciphertext[l - 27] == 'i' && 
           ciphertext[l - 26] == 'v' && 
           ciphertext[l - 25] == '='
}

The migration over there has three steps:

1. Everybody decrypts both nip 04 and nip 44
2. Everybody starts encrypting with NIP44 in their own timeline.
3. Everybody drops support for encrypting and decrypting with nip04.

[jklein24]

Ack

FYI, NIP-46 (#1248) is just dynamically figuring out if the encryption is nip-04 or nip-44.

fun isNIP04(ciphertext: String): Boolean {
    val l = ciphertext.length
    if (l < 28) return false
    return ciphertext[l - 28] == '?' && 
           ciphertext[l - 27] == 'i' && 
           ciphertext[l - 26] == 'v' && 
           ciphertext[l - 25] == '='
}

The migration over there has three steps:

1. Everybody decrypts both nip 04 and nip 44
2. Everybody starts encrypting with NIP44 in their own timeline.
3. Everybody drops support for encrypting and decrypting with nip04.

Thanks, this also makes sense as the alternative path for NIP44 migration, and is certainly simpler. My main reason for not going this route right now is so that connections can use NIP44 right away if both sides indicate support, without needing to wait for a migration or try encrypting with NIP44 first, getting an error, and then re-encrypting with NIP-04.

I also think in general a versioning scheme will be useful for NWC for future breaking changes if they're needed, but I recognize that it adds complexity.

[rolznz]

Linking the notifications PR which ideally is merged (it has multiple implementations using it already) and has implications for versioning.

Because there is no communication from the client application to the wallet service to listen to notifications (the client application is simply subscribing to the relay), I think the simplest way would be to update the notifications PR to use a different notification event kind for version 1.0, and start including the tag in the notification event. That way, version v0.0 apps will not break by trying to decode v1.0 notifications.

I do not know how useful the version tag will be on notifications though, or if another breaking change in the future would require another change to the notification event kind, since the wallet service might still want to support the previous version.

[ekzyis]

Looks good!

The v tag leaks metadata about connections but it also allows us to track adoption rate of new versions and thus when we can start dropping old versions.

Is this implemented in any wallet service yet with which I could test against as a client?

[ekzyis]

What about patch versions? I think clients would also be interested in knowing if the service has the latest patches.

[jklein24]

My take was the same as @bumi mentioned with regards to patch versions. If they don't really change compatibility, and versions don't bump often, I think patch versions will add more complexity then benefit.

[bumi]

lgtm, thanks!
I do not expect many versions (and rather bigger changes like the switch to nip-44) because too many versions (of a spec) lead to incompatibilities which must be avoided. (thus I do not care about patches here)

Alby plans to implement this with NIP44

[jklein24]

Looks good!

The v tag leaks metadata about connections but it also allows us to track adoption rate of new versions and thus when we can start dropping old versions.

This is a really good callout, thanks for pointing it out. The adoption rate tracking is nice though. I think this is a reasonable tradeoff.

Is this implemented in any wallet service yet with which I could test against as a client?

Currently only in a demo with the uma-nwc-server docker image, but it sounds like it'll be in alby hub in the not-so-distant future too :-)

[jklein24]

Linking the notifications PR which ideally is merged (it has multiple implementations using it already) and has implications for versioning.

Because there is no communication from the client application to the wallet service to listen to notifications (the client application is simply subscribing to the relay), I think the simplest way would be to update the notifications PR to use a different notification event kind for version 1.0, and start including the tag in the notification event. That way, version v0.0 apps will not break by trying to decode v1.0 notifications.

I do not know how useful the version tag will be on notifications though, or if another breaking change in the future would require another change to the notification event kind, since the wallet service might still want to support the previous version.

Thanks for flagging this! I defer to you on what's best for notifications and how versioning could be relevant for them. What you've suggested with a new notification event kind sounds somewhat reasonable to me for forward-compatibility, but I could see how it'd be a bit of an annoying migration flow :-/

[ekzyis]

This is a really good callout, thanks for pointing it out. The adoption rate tracking is nice though. I think this is a reasonable tradeoff.

I realized that it actually doesn't leak anything about connections since the requests and responses use ephemeral events (or are at least supposed to) so we can't track adoption rate of clients. We can only track adoption of wallet services by filtering for kind:13194. I think knowing which wallet service supports which version is unrelated to privacy unlike being able to distinguish individual connections from each other.

Was just something I noticed and wanted to mention, it wasn't (and still isn't) a real concern of mine.

I'll make sure that SN supports NIP-44 for NWC soon to see if I have any other feedback.

[frnandu]

Notifications got merged , #1164
so here it needs to adapt to new spec, conflicts resolved, and there is the issue of which kind of encryption to use on kind =23196 notifications, which might be subscribed by different client with same nostr+wallet:// connection that support different versions (nip04 or nip44) to be discussed...

[jklein24]

Notifications got merged , #1164 so here it needs to adapt to new spec, conflicts resolved, and there is the issue of which kind of encryption to use on kind =23196 notifications, which might be subscribed by different client with same nostr+wallet:// connection that support different versions (nip04 or nip44) to be discussed...

Apologies for the slow response here, thanks for poking on it @frnandu. @rolznz I'm curious whether you have thoughts on the right path here wrt notifications. I can update to use a new notification event kind for v1 like you suggested and we can just use nip44 to encrypt those as well as add the version tag. 23196 events would still need to be nip04-encrypted though in that case. In addition, I guess if a wallet service supports both v0 and v1+, it might need to send both notification even kinds for a given event until it knows which version the client supports (via the client's most recent nwc request). As an example:

1. Client establishes a connection.
2. Wallet receives a payment and needs to send a notification, so it sends both 23196 (nip04) and 23197 (nip44) events to the client.
3. Client makes some request (get_info) which indicates a v tag of 1.x. The wallet service can optionally cache this version as its notification version as well, so that it only needs to send one event to that client next time. Note that there's some risk here in that clients need to make some request after upgrading their version in order to receive the right notification type. However, I think in practice, it's pretty likely that this would happen anyway.

The extra per-client-version caching does add some complexity on the wallet side, which is unfortunate. However, since wallets already need to cache info about clients (permissions, budgets, etc), this seems pretty manageable. It would also be good to have some way for clients to reliably dedupe 23196 and 23197 events if they have to listen for both at first. I guess for the existing notifications, the payment hash is sufficient. If we don't add any more notification types before v1, maybe that's good enough?

Does that sound right to you?

[rolznz]

@jklein24 that sounds good to me, except the below, I am not sure is necessary:

The wallet service can optionally cache this version as its notification version as well, so that it only needs to send one event to that client next time. Note that there's some risk here in that clients need to make some request after upgrading their version in order to receive the right notification type. However, I think in practice, it's pretty likely that this would happen anyway.

I hope we can convince client apps to update within 6-12 months, and then stop sending NIP-04 notifications from wallet services after that. As far as I know, Alby Hub is the only wallet service that supports sending NIP-47 notifications right now (although there are multiple clients that consume them). As long as we update Alby Hub to support both, client apps are free to switch to NIP-44.

[ekzyis]

Regarding SN: we will rely on NDK to handle NIP-04 vs NIP-44 (stackernews/stacker.news#1590) but it doesn't use NIP-44 for NWC yet.

There is a ticket though (nostr-dev-kit/ndk#166) and two related PRs (nostr-dev-kit/ndk#233, nostr-dev-kit/ndk#279) but not sure if they will include NIP-44 for NWC. I'll ask.

[frnandu]

3. Client makes some request (get_info) which indicates a v tag of 1.x.
   The wallet service can optionally cache this version as its notification version as well, so that it only needs to send one event to that client next time.
   Note that there's some risk here in that clients need to make some request after upgrading their version in order to receive the right notification type.
   However, I think in practice, it's pretty likely that this would happen anyway.

Mostly yes, but a NWC connection can be used by multiple clients, even if that is not the recommended way, I sure do it often.
And each client might support different versions, so assuming that a connection is tied to a strict client version support might not always be the case.
Worst UX scenario is for example a case where notifications work for some legacy client, but then you use that same connection in a newer supporting-nip44 client and the wallet service decides that from now on notifications for that connection will only send in new 23197 kind, and the legacy client stops receiving notifications in 23196.

[jklein24]

3. Client makes some request (get_info) which indicates a v tag of 1.x.
   The wallet service can optionally cache this version as its notification version as well, so that it only needs to send one event to that client next time.
   Note that there's some risk here in that clients need to make some request after upgrading their version in order to receive the right notification type.
   However, I think in practice, it's pretty likely that this would happen anyway.

Mostly yes, but a NWC connection can be used by multiple clients, even if that is not the recommended way, I sure do it often. And each client might support different versions, so assuming that a connection is tied to a strict client version support might not always be the case. Worst UX scenario is for example a case where notifications work for some legacy client, but then you use that same connection in a newer supporting-nip44 client and the wallet service decides that from now on notifications for that connection will only send in new 23197 kind, and the legacy client stops receiving notifications in 23196.

That's another really good point, @frnandu. I guess we're stuck with what @rolznz is suggesting for now then - we just always send both events and clients will need a way to dedupe (payment hash for now). @rolznz do you have a sense of how broad client usage is for notifications at the moment? I'm trying to get a rough gauge on how painful/long the migration will be so that hopefully we don't need to be firing both notification events for long. The client-side deduping is kinda gross :-/.

[rolznz]

@jklein24 I would not do client-side deduping, just use the get_info call first to figure out if the wallet service supports NIP-44 or not, and then only subscribe to the correct kind, like you suggested in 3. Or do you see issues there?

I'm not sure how long the migration would take, but as soon as we release a version of Alby Hub that supports it I would contact all known developers and ask them to update. The apps I know about are listed in #1164 (comment)

We are actively working on supporting NIP-44.

[jklein24]

@jklein24 I would not do client-side deduping, just use the get_info call first to figure out if the wallet service supports NIP-44 or not, and then only subscribe to the correct kind, like you suggested in 3. Or do you see issues there?

I'm not sure how long the migration would take, but as soon as we release a version of Alby Hub that supports it I would contact all known developers and ask them to update. The apps I know about are listed in #1164 (comment)

We are actively working on supporting NIP-44.

Sorry for the delay here, but I've rebased and added proper context for notifications @frnandu @rolznz. Let me know if this looks right to you.

[rolznz]

Would it also make sense to propose v as a standardized tag (further down in this file)?

[jklein24]

I actually considered this, but decided to avoid the larger discussion on a general version tag or implications in nostr events more broadly than in nwc. That being said, maybe it's worth listing here to be explicit. @vitorpamplona any thoughts on adding the v tag here as a standardized tag?

[vitorpamplona]

Better to stay out of it since NIP-47 seems to be the only place using it. Let's see if other nips decide to adopt first.

[kumulynja]

Nack at first sight, I think adding versioning like this will make contributors lazy and make them think less about backwards-compatibility since their new update can just get a new version assigned. This in the end may cause a lot of different versions between clients and wallets, and cause a lot of compatibility problems for the users or have the devs still need to implement every possible version. I think updates to the protocol should be backwards compatible and optional as most as possible, if they are not, I prefer it can be known from the Connection or Auth URI, so that wallets or clients know if they are compatible or not before the connection is confirmed, instead of changing the events and adding new event types.

Update:
I read up on some of the concerns and see the initial thought was also doing it via the URI, but updating would be a problem then, I agree updates should not be interactive. Will give the proposal and my initial remarks some more thought.

[ekzyis]

I think adding versioning like this will make contributors lazy and make them think less about backwards-compatibility since their new update can just get a new version assigned.

Why do you think it would be so easy to get a new major version merged as a NIP?

If a NIP is bad, devs won't implement it. Anyone can propose anything, but it's not their decision if it will get adopted. You basically explained it yourself here:

This in the end may cause a lot of different versions between clients and wallets, and cause a lot of compatibility problems for the users or have the devs still need to implement every possible version.

[kumulynja]

Why do you think it would be so easy to get a new major version merged as a NIP?

I don't know that, just as you don't know it will not be. But it is in people's nature to take the easy and fast route if it is there instead of a possible better -not that obvious/easy- route.

If a NIP is bad, devs won't implement it. Anyone can propose anything, but it's not their decision if it will get adopted. You basically explained it yourself here:

Who is speaking about bad NIP's? We are not talking about creating a new NIP here, in that case we wouldn't need versioning. We are talking about changing a NIP, and I think compatibility problems is one of the main issues with that and why I think this shouldn't be taken lightly.

[kumulynja]

Another concern I have with versions like this instead of just optional add-ons/changes that are backwards compatible or can be signalled for individually is that I might like one thing of version y, but dislike another thing and prefer to stay with the version x way for that other thing. With versioning I don't think this flexibility is possible, you either upgrade to version y completely or you stay on version x completely, no in between. I prefer a way that upgrades/changes can be picked individually, if there are no dependencies between one another of course.

[jklein24]

Nack at first sight, I think adding versioning like this will make contributors lazy and make them think less about backwards-compatibility since their new update can just get a new version assigned. This in the end may cause a lot of different versions between clients and wallets, and cause a lot of compatibility problems for the users or have the devs still need to implement every possible version. I think updates to the protocol should be backwards compatible and optional as most as possible, if they are not, I prefer it can be known from the Connection or Auth URI, so that wallets or clients know if they are compatible or not before the connection is confirmed, instead of changing the events and adding new event types.

Update: I read up on some of the concerns and see the initial thought was also doing it via the URI, but updating would be a problem then, I agree updates should not be interactive. Will give the proposal and my initial remarks some more thought.

I see your update and am glad you're rethinking things. I'll just note that I'm totally open to other suggestions to update NWC to NIP-44 if you see another path. To me, versioning is useful for breaking changes, but should be used very sparingly only for breaking changes like this one. I agree with your concerns about striving to make things backwards-compatible in every case possible, so if you see a way to do that here for NIP-44, I'm all ears.

[kumulynja]

I see your update and am glad you're rethinking things. I'll just note that I'm totally open to other suggestions to update NWC to NIP-44 if you see another path. To me, versioning is useful for breaking changes, but should be used very sparingly only for breaking changes like this one. I agree with your concerns about striving to make things backwards-compatible in every case possible, so if you see a way to do that here for NIP-44, I'm all ears.

Non-breaking changes >> breaking changes breaking things and getting them fixed fast >> versioning complexity and inflexibility

Saying it "should" be used sparingly only for breaking changes is like politicians saying tax payers money should be used sparingly and it will only be used for the most in need. No guarantees and once it's there, it will be used inappropriately too (maybe not by current contributors/maintainers, but future ones).

I am working on a proposal for nip44 migration though, hope I can figure out the details, make sure things check out and share it by next weekend.

[kumulynja]

@jklein24 wrote out a quick initial draft that also explains how to know which encryption to use: master...kumulynja:nips:client-info-event

The main advantage it has over strict versioning is the flexibility to be able to pick and choose different features with optional fields/tags instead of being bound to a strict set of all features of a single version.
And it also doesn't require version tags on all requests, neither do I think a new notification kind is needed. Everything should be clear for both sides from just the Info Events, which can be used to dynamically update connections in the future too.

[rolznz]

Ack (implemented by Alby Hub, Alby Go, Alby JS SDK, http-nostr)

Hopefully we remove the old notification event within 6 months - 1 year once remaining projects update to use NIP-44. CC @frnandu @Kukks

[rolznz]

@kumulynja I think the client info event increases complexity as it requires the client to send this event and the server to maintain states about different client capabilities. I also think client key rotation is unnecessary as a user can delete the connection from their wallet if they suspect that particular connection is compromised.

[kumulynja]

@kumulynja I think the client info event increases complexity as it requires the client to send this event and the server to maintain states about different client capabilities. I also think client key rotation is unnecessary as a user can delete the connection from their wallet if they suspect that particular connection is compromised.

It is just one event, which is similar to the wallet side Info Event already, so pretty understandable I think. And introducing just one event gives three big improvements: flexible capabilities negotation (implicit versioning), anti-phishing and non-interactive client key rotation. If later we will still need to solve those things separately, we will need to introduce it anyway, or find three other ways, which probably becomes more complex.

About the server needing to maintain state, well not really, it can always just fetch the Info Event to know if it doesn't want to store it locally. But not really sure what you mean with "states about different client capabilities" on the server? Do you mean on the wallet service side or a server of the client side? And do you mean you prefer the requests having a tag so the wallet service side doesn't have to look up anything anymore? If that's the case, I prefer adding an "encryption" tag in the request instead of a version.

I think if we really want to be able to say that NWC will be the payment protocol of the future and have big companies use it, things like phishing protection and key rotation can not be ignored. And it is very difficult for a user to know if a connection is compromised until all his money (budget) is gone one day, the idea is that this is non-interactive. The client will probably know sooner or be able to act faster when it suspects any compromise, so that the client can rotate its own client keypair makes more sense to me than the user having to delete the connection manually (when it is probably too late already) and then have to add it again as well.
And the Client Info Event and client key rotation and all other things described in the draft are all optional btw, no one needs to implement it if they think it is too complex or they don't consider it valuable. That's the good thing about it. It can be done, it's not necessary, all optional and backwards compatible.

I really think adding versioning like this is a mistake and adds more complexity than just one event. And I definitely think the inflexibility concern needs to be addressed before merging this. Supporting a version will require implementing the new version completely and will make it impossible to pick-and-choose features individually. And you will basically lose staying backwards compatible as versions in this way may introduce breaking changes and a fragmented ecosystem.

I would really like to ask to wait to see a working implementation and a better proposal write-up of the Client Info Event first to analyse and compare the "complexity" of both, before making a decision, as I think this is a pretty important update with no way back afterwards (because of precipitating breaking changes).

[kumulynja]

To give another example:

Let's say the latest version of nip47 is 2.5 and a client or wallet is still on 2.1, but it needs just one new addition for their use-case that happened to be included and labeled as part of 2.5. Because of this explicit versioning, they can not just implement and signal for this one new addition individually. They have to implement ALL features of 2.2, 2.3, 2.4 and 2.5 as well just for this one addition they are actually interested in, just to follow the versioning and not cause any other incompatibility issues by signalling they are v2.5 with only the addition of interest actually added.

Or do you guys not like the fact, in Nostr itself for example, that you can just add NIPs individually without having to comply with a specific Nostr version?
NWC is an open protocol on its own that should be seen in the same way as Nostr itself concerning "versioning" or feature additions. It is not a software implementation on a specific version.

[vitorpamplona]

There is always some versioning going on. It can be implicit or explicit, but NIPs always evolve in unpredictable ways that require some form of versioning. While I agree that versioning the entire NWC in one number is bad, it is likely that every NIP will need some form of full-coverage versions from time to time.

But versioning, to me, either explicit or implicit, just supports a temporary transition state for current implementers. The key is deprecating old versions as quickly as possible to clean up the implementation space for newcomers.

It is still very early stages for Nostr and NWC. We can do many things now that we will not be able to in the future. So, let's make sure that if we need to break things, we break them right now and move on quickly.

[kumulynja]

While I agree that versioning the entire NWC in one number is bad

So we agree that explicit versioning for NWC is bad 👍

It is still very early stages for Nostr and NWC. We can do many things now that we will not be able to in the future. So, let's make sure that if we need to break things, we break them right now and move on quickly.

I agree that it is still very early stage and better to break things now if we have to. But, the thing is that just to add NIP-44 encryption, we don't have to break anything. And we don't need explicit versioning. Just adding an "encryption" tag does the job as good as linking it to an explicit version, without the many downsides of explicit versioning for decentralized protocols as mentioned.
So, there really is no need to introduce versioning now. And I would use the fact that it is still very early stage to not unnecessarily introduce something that inflexible and opinionated this early on if there is really no need for it.

[jklein24]

After chatting through this more, I've whipped up a quick alternative PR which implements the simplest possible alternative like @kumulynja is suggesting. I think this is a pretty reasonable alternate path for the nip44 upgrade if folks prefer that route, but at least this can help us compare the difference.

In general, I don't totally agree that a version number is a bad thing for the future if there are going to be actually breaking changes, but I do agree that we should only use that heavy method if we absolutely have to. In this case, it does seem like an encryption tag might do the trick. Let me know what you all think.

[rolznz]

@jklein24 @kumulynja this alternate idea seems like a better option and more consistent with how we indicate supported methods and notifications. I will see if we can update our projects to support the encryption tag and for a short time keep the version tag for backward compatibility.

[jklein24]

@jklein24 @kumulynja this alternate idea seems like a better option and more consistent with how we indicate supported methods and notifications. I will see if we can update our projects to support the encryption tag and for a short time keep the version tag for backward compatibility.

I'll swap UMA Auth over as well. Should be a pretty quick change.