NIP-38 - Encrypted Group Chat using a single shared secret

[vishalxl]

NIP-38 is modeled on NIP-28, group chat. Main difference is use of a shared secret, and using p-tags to mention members in channel create and update events. Simplicity is one of the main goals.

Implementation for the events mentioned ( 140-142) is in Nostr Console 0.0.9 beta release.

Note 1: One important aspect of this NIP-38 ( kind 14x) is that they have a participant list. The 4x group chat is public by design, and it is unlikely that it will have a participant list anytime in future ( though it can add a PoW condition to participate easily). So the 14x public chat, whether it has secret shared secret, or even in case where its 'shared secret' is public, can still serve well because it allows the channel creator to mention as p-tags in 140/141 events who are the participants of the group.

Edit: pull request made more for discussion.

[jb55]

we really shouldn't do this over kind 4. this is really hacky. ideally it would be a protocol message that clients can handle without parsing, display an invite properly, etc. Users shouldn't have to see a shared secret at all.

[vishalxl]

Agree with this that there is no good reason to use kind 4 here.

Invite suggestions on what kind it should be. I think it would be good to have one special kind where different apps can store or send their information from where they can pick it up. So that user can know what app is sending what message.

Edit: Kind 104 would be a good number. Its 100 away from 4, and no one has yet used such messages.

Also then the format of the message, or its 'header' or first few words etc, should help differentiate between different apps.

Currently I have used:

App Encrypted Channels: inviting you to encrypted channel <channel id> encrypted using private public keys <shared private key> <corresponding shared public key>

So it envisages a format like

"App" <app name>: <app specific information>

[vishalxl]

Edited the proposed NIP 38 and mentioned kind 104 rather than kind 4 to communicate the shared-secret.

[vishalxl]

There is not much parsing involved for shared-secret message. Once the app matches the first few words, which are "App Encrypted Channels: ", then the required information is at well-known points in the message. If strInvite is the given message as a string then the required values are always at the same place:

Encrypted Channel id : strInvite.substring(58, 58 + 64)
Secret private key : strInvite.substring(159, 159 + 64);
Secret public key : strInvite:substring(224, 224 + 64);

And the length of strInvite should be 288 chars.

[vishalxl]

I now think the above message should ideally be in json format. will work on it further in that direction.

Unless you have some other idea there?

[earonesty]

no need for public key, it can be derived

i'll throw up a nip for "more secret" dm's

[jb55]

Something me and @Semisol talked about awhile back was creating a new shared secret whenever a participant is removed, and this is encrypted into tags for each participant.

The client would then query all the update messages and add all of the distinct keys to their decryption key set. Then for older messages it would try to decrypt with each key in the set until a successful decryption.

This would enable you to re-key channels once participants are removed with a pretty simple algo. Also you don't need to send invites in this case either, the key is automatically encrypted to the user in the channel creation/update tags.

[arejula27]

Could be a good option to create a new shared secret when a new user is added, maybe someone want to add user for the future messages but dont want to allow them to read the previous ones

[vishalxl]

Removing of participant ( or cases where a new secret is needed for all ) can be catered to by this method, where we allow the creator (or anyone for that matter) to "fork" a group, so that the new fork will have a new id, new secret. We mark the new group as a 'fork' by adding some specific tag info.

Then internally in clients, user can be seamlessly shown messages from what are two channels ( if you go by channel id itself), but are logically the same cause the client knows which channels are connected logically as same.

This is the simplest way I think we can implement removal: it also has the logical consistency ( which will make coding easier imo), that each channel id is only ever connected with one shared secret. But using extra tags, we can establish a logical connection ( that 2+ channels are same channel as far as user knows).

This even allows some member to fork off a new group; people may want to do it. Lets say admin is absent, so they move to a new 'channel id' , but logically clients will/may show them just one group. Plus, if they want, they can continue using the same shared secret; there is no need for anyone to enforce that different shared secret has to be used; but good clients will obviously use different shared secret on removal of a member(s).

[earonesty]

im building this now, based on the "considered harmful discussions" with @paulmillr mostly as a POC. using the new "versioned encryption" system, ephemeral public invitations and well-known group public/private key, like in this document. working fine so far. you can play with a test flight version soon.

https://github.com/ArcadeLabsInc/arclib/pull/20/files

[vishalxl]

A related note. We may need to have in total three types of "channels":

a) totally public kind 4x channels
b) encrypted channels with membership (like this proposed NIP 38 with kind 14x)
c) public channels with membership ( where content is public but only allowed people are considered in), could be kind 24x

Edit: On second thought, this needs revisiting. Cause membership is possible even in channel 40 with an extension of adding p tags as members.

[Semisol]

c) public channels with membership ( where content is public but only allowed people are considered in), could be kind 24x

The state for this shouldn't use replaceable events: you want to see who the historical members were to get their events. Another issue: backdating

[cmdruid]

IME, encrypted group chat is a rabbit-hole of complexity.

A lot of folks use double ratcheting as a standard. All the requirements are found in WebCrypto:
https://en.wikipedia.org//wiki/Double_Ratchet_Algorithm

I have seen this method scale (and fail) gracefully on rooms with 100+ members. So maybe that is a good option.

Otherwise, I would suggest a protocol that uses forward secrecy by default, so that updating the shared secret is easy. A chat room has to broadcast frequently, so updating the secret is no big deal. Everyone has to agree on the state of channel members though.

I also think that if there are plans to use group encryption and membership, you may as well add some role / admin features as well. Matrix uses a concept of "power levels" that works really well.

Anyway, my 2 sats on the discussion. :-D

[ghost]

Thanks for writing this NIP. I think it can be used as an alternative to #49 for joinstr.

I have not tested it yet in nostr console. Will do it today.

[fiatjaf]

I think this may be a superior alternative, since it would be fully stateless like NIP-04: #72 (comment)

[ghost]

@fiatjaf

I think this may be a superior alternative, since it would be fully stateless like NIP-04: #72 (comment)

Can you explain the stateless part in detail or what are the advantages of this method over the one shared in this PR?

@vishalxl

The meta-data for any message is practically much public in nostr. So such group chat should not be considered private. Anybody can see who is sending message(s) to what channel, and when.

This might not be perfect for all use cases however it should work for some. As long as shared secret and messages are private it will work for 2 of my projects: joinstr and p2p exchange

Not sure if any other client and library has already implemented it apart from nostr-console.

Once members are 'removed' by using the kind 141 event for a channel, they can continue to read the new messages posted in the group, because currently there is no provision to change the shared secret of a single group. The remaining users, may howerver choose to create a new group where the recently-removed member is not a member, and which would have a new shared-secret.

This should not be a concern if clients implement it correctly and create a new group chat once someone leaves with the rest of the members.

[fiatjaf]

Can you explain the stateless part in detail or what are the advantages of this method over the one shared in this PR?

Each event contains its own decryption keys, just like NIP-04, that is what I called "stateless".

[vishalxl]

aB, aC and aD

The three secrets that Alice generates in that scheme you mentioned @fiatjaf , I can't tell what happens to them, in that where do they go after steps 4 and 5, as they are not mentioned after 5? I am having trouble understanding what is happening here.

So in step 4 Alice generates three secrets, one each for 3 receivers. In step 5 she uses them to encrypt x. Then Later are those secrets never needed by Alice or any other people? Since x was encrypted with those secrets, wont' they be needed again to decrypt x? And if they will be needed, by whom are they needed? Are they given to respective recipients?

[cmdruid]

What about a replaceable event that simply provides each member with the latest shared secret for a channel, encrypted to their pubkey?

If you are a member on the list, then there will be a shared secret available for you to decrypt. Otherwise, you get nothing.

Whenever the channel owner wants to update the member list, they simply update the event with a new shared secret, and update the encrypted keys for each member. Members can subscribe to the replaceable event to get the latest key.

You could also throw other channel metadata in there as well, just use the whole content field as a metadata store for the channel.

[vishalxl]

What about a replaceable event that simply provides each member with the latest shared secret for a channel, encrypted to their pubkey?

Pretty much this is what is being done in this NIP 38, where the single shared secret is being shared using kind 104, which is encrypted to the recipeients respective pubkey. So member m1 gets from creator C a kind 104 which is like a DM, which mentions the shared secret. Now whether that's replaceable or not is up for discussion.

Whenever the channel owner wants to update the member list, they simply update the event with a new shared secret, and update the encrypted keys for each member. Members can subscribe to the replaceable event to get the latest key.

Updation is propsed differently - the NIP 38 proposes that a single channel is always only associated with a single secret. To remove members, for example, a new channel is created with new secret. This keeps the logic simple , as in one channel, one secret, always.

You could also throw other channel metadata in there as well, just use the whole content field as a metadata store for the channel.

I am thinking of having a json for kind 104, with details about group also thrown in, if that's what you mean too.

[cmdruid]

Pretty much this is what is being done in this NIP 38, where the single shared secret is being shared using kind 104, which is encrypted to the recipeients respective pubkey. So member m1 gets from creator C a kind 104 which is like a DM, which mentions the shared secret. Now whether that's replaceable or not is up for discussion.

Can you elaborate on this more? Are you encrypting the private key using the public key and ECDH?

[cmdruid]

Is it something like this:

Both Parties

// Generate keypair for channel owner and channel member.
const { ownerPubKey,  ownerPrivKey  } = KeyPair.generate(32)
const { memberPubKey, memberPrivKey } = KeyPair.generate(32)

// Both parties compute shared secret between owner <-> member.
const sharedSecret = 
     ECDH(memberPubKey, ownerPrivKey)
  || ECDH(ownerPubKey, memberPrivKey)

// Use shared secret to derive a shared keypair.
const { sharedPubKey, sharedPrivKey } = KeyPair.from(sharedSecret)

Channel Owner

// The channel owner create a secret for the channel.
const channelSecret = Random.bytes(32)

// To share the channel secret with a select member,
// owner encrypts the secret using sharedPubKey
const encryptionKey = ecdh(ownerPrvKey, sharedPubKey)
const encryptedString = encrypt(encryptionKey, channelSecret)

// Use tag field as a hash map for distributing member keys.
const memberTag : HashMap = [
  sharedPubKey,    // Hex-encoded compressed public key.
  encryptedString  // AES-encrypted using member key and ECDH.
]

// Example of a channel event.
const channelEvent = {
  id      : 'eventId',
  kind    : 10000,
  tags    : [ memberTag ],
  content : encryptedMetaData,
  pub     : ownerPublicKey,
  sig     : ownerSignature
}

Channel Member

// Check if shared pubKey exists in tags hash map.
if (channel.tags.has(sharedPubKey)) {
  const encryptedString = tags.get(sharedPubKey)
  // Use private key to derive secret between shared <-> owner.
  const encryptionKey = ecdh(sharedPrvKey, ownerPubKey)
  // Decrypt channel secret using shared encryption key.
  const channelSecret = decrypt(encryptionKey, encryptedString)
  // Decrypt the channel meta data using the channel secret.
  const channelMetaData = decrypt(channelSecret, encryptedMetaData)
}

[vishalxl]

@cmdruid that's pretty much the outline of this NIP 38 in code/pesudo-code. I have added some mention of kind 104, and otherwise edited to make it more in sync:

Both Parties

// Generate keypair for channel owner and channel member.
const { ownerPubKey,  ownerPrivKey  } = KeyPair.generate(32)
const { memberPubKey, memberPrivKey } = KeyPair.generate(32)

// Both parties compute shared secret between owner <-> member.
const sharedSecret = 
     ECDH(memberPubKey, ownerPrivKey)
  || ECDH(ownerPubKey, memberPrivKey)

// Use shared secret to derive a shared keypair.
const { sharedPubKey, sharedPrivKey } = KeyPair.from(sharedSecret)

Note: this shared secret is also used in kind 4 messages.

Channel Owner

When creating the channel, the channel creator will create a random 'shared secret' and then create two messages

a) First is a kind 104 message using the following logic, which is sent to EACH of the members ( it contains the shared secret, encrypted using the newly generated key mentioned in part "both parties")

// The channel owner create a secret for the channel.
const channelSecret = Random.bytes(32)

// To share the channel secret with a select member,
// owner encrypts the secret using sharedPubKey
const encryptionKey = ecdh(ownerPrvKey, sharedPubKey)
const encryptedString = encrypt(encryptionKey, channelSecret)

// cmd: Use tag field as a hash map for distributing member keys. 
// my comment: the hashmap is just application logic; this can be the creator going in a loop. The application will have generated different secret for each intended member/recipient, so it will send them to them using their specific secret. This is the kind 104 , which is modeled on kind 4.
const memberTag : HashMap = [
  sharedPubKey,    // Hex-encoded compressed public key.
  encryptedString  // AES-encrypted using member key and ECDH.
]

b) second is a kind 140 event with following format, which also has member tags so people can fetch it with #p REQ:

// Example of a channel event.
const channelEvent = {
  id      : 'eventId',
  kind    : 10000,
  tags    : [ memberTag ],
  content : encryptedMetaData,
  pub     : ownerPublicKey,
  sig     : ownerSignature
}

Note: encryptedMetaData is in specific 'one line' format in this NIP 38 as of now, but should ideally be in json to add more information.

Channel Member

// Check if shared pubKey exists in tags hash map.
if (channel.tags.has(sharedPubKey)) {
  const encryptedString = tags.get(sharedPubKey)
  // Use private key to derive secret between shared <-> owner.
  const encryptionKey = ecdh(sharedPrvKey, ownerPubKey)
  // Decrypt channel secret using shared encryption key.
  const channelSecret = decrypt(encryptionKey, encryptedString)
  // Decrypt the channel meta data using the channel secret.
  const channelMetaData = decrypt(channelSecret, encryptedMetaData)
}

NB I am assuming kind 4 works like this ECDH logic you mentioned, without verifying. Because kind 104 follows just how kind 4 does.

[cmdruid]

NB I am assuming kind 4 works like this ECDH logic you mentioned, without verifying. Because kind 104 follows just how kind 4 does.

Yeah you are correct, ECDH meaning Elliptic-Curve Diffe-Hellman, used to derive the shared secret. NIP-04 uses that and then encrypts using AES with random IV.

Does the current proposal send DMs to all members on each channelSecret update? If that's the case, would it scale better for members to subscribe to the channel memberlist for updates, while using DMs sparingly to send out invites?

Also side note: Since interim sharedPubKey obfuscates the real participating pubkeys, I don't believe the memberlist has any sensitive information? I think it would be neat if channelEvents are used to set up an encrypted group chat, but without it being publicly linked to the group chat itself, or any of its members. Then it would simply be a rendezvous for members to get the current state of the encrypted group.

I like to use a hash of the channelSecret to tag the group chat, as publishing the hash does not reveal the secret, and members can subscribe to the tag as a group feed. You could also use the encryptedMetaData to store the real list of member pubkeys, so the sharedKeys aren't used outside of delivering the channelSecret.

[vishalxl]

Does the current proposal send DMs to all members on each channelSecret update? If that's the case, would it scale better for members to subscribe to the channel memberlist for updates, while using DMs sparingly to send out invites?

DM's ( of kind 104) need to be sent only when the secret is shared. Then no DM's need to be sent.

I like to use a hash of the channelSecret to tag the group chat, as publishing the hash does not reveal the secret, and members can subscribe to the tag as a group feed. You could also use the encryptedMetaData to store the real list of member pubkeys, so the sharedKeys aren't used outside of delivering the channelSecret.

So then this will happen: creator gives shared secret key in DM to members. From that the members can derive its corresponding public key, and when they want to send messages to the group, they use that 'public key' as an e tag, lets call it channel-ID-tag.

So when a member m1 sends a group chat message, all that outsiders will see is some random e tag is attached to the message. They would have no idea what that e tag is. They will also see other people using that e tag, so people can guess its one group.

And since the member list is published as encrypted data, there is no publicly-seen member list. We can further improve it thus: there can be MULTIPLE channel-ID-tag, and the group would be told, as encrypted message, all the channel-ID-tag's of a group. The members can send message using any of the tags, and all the receivers can fetch all the tags. And since the set of channel-ID-tag for any channel was encrypted from start, no one outside would have any clue who is messaging as a group, as long as enough number of channel-ID-tag 's are used for each channel.

Along with encrypted membership list as you mentioned too ( which can be updated with 141 kind as encrypted data too on addition of members or renaming of channel), this would give a lot of 'privacy' to users. Or some amount of anonymity, compared to the very base case scenario.

NB. This logic can be used to create 1-1 communication channels between two people too, to have enhanced privacy over kind 4 DM's. They will use the first do a kind 104 DM to exchange these per-channel tags, and then communicate to each other using those tags. This increases the privacy of communication, because after the first DM, no one will know who is talking to whom ( except by some meta analysis, which can never be confirmed, unlike the open meta data of kind 4 messages).

[itslenny]

What if a user wants to remove themself? Obviously the client can just ignore messages, but it seems like a good client should also notify the relay that they left the room, and the admin should remove them from the list and roll the keys.

[earonesty]

that's a good idea

[Semisol]

Metadata leaks?

[earonesty]

yes, we can avoid this by using ephemeral public keys listed here:#570 (comment)

[Semisol]

Metadata leaks? Also you need to separate tags with ,.

[earonesty]

tags should be in the envelope, yes - for a really private room. there is a need for both (hey whole world, this is a private room, please beg for an invite)

[earonesty]

see also: #570 (comment) for a nice replacement that encrypts everything in a channel

[Semisol]

Ideally a new shared secret can be generated when participants are removed.

[earonesty]

equivalent to rolling a new room and saying that's where we're talking now

[Semisol]

Please use replaceable events.

[itslenny]

This part could handle encryption the way it is done on #168 instead. This creates a single DM that can be read by multiple people by encrypting it with a unique secret that is included in the tags for every recipient.

This avoids the channel owner needing to send N messages every time someone is removed from the group.

Alternatively, and I don't think this is a good idea, but sharing it in case it sparks a better idea... this shared secret part could be removed entirely and all of the messages could be sent using the proposed NIP48 method. The issue with this is that the individual messages would have to include the entire list of people that can decrypt and bad clients could technically add/remove people from that list when sending messages and not obey the kind 141 participant list

[earonesty]

ok, just read that. sending N messages is avoided but you still need to do N encryptions. there's not a big difference really, except now the message is huge. so when someone wants to look for their message, they get a bunch of keys they can't use. i like the encryption in 38 better, yes you have to send more "messages", but they're smaller and this is better for routing, and you can send to different relays, etc.

[itslenny]

This message feels easy for humans to read and hard for machines to read. We should optimize for the opposite. Also, the public key can be derived from the public key so it does not need to be included.

My suggestion would be content="<shared private key>"

Then, include the as a tag of some sort.

[earonesty]

this should not use a nip04 style encryption, instead it should use directional encryption

encrypt({
sender pub,
sender sig of this payload,
content=....
tags=...
}, ephemeral pub),

no tags outside the envelope, pubkey is ephemeral. recipient verifies the sender's provenance after decryption

all messages should work this way in this protocol - except possibly for a public broadcast advertisement (if you want everyone to know about the room)

[earonesty]

#570 (comment)

[guilhermegps]

Any news about it?

[MTG2000]

What's the state of this NIP??
The last update is a couple of months ago.

So is this still relevant?

[vishalxl]

Status as of May 2023: This is implemented, the way its mentioned right now, in nostr console.

There are some improvements needed, or suggested, such as using json format to share initial shared secret etc, and there are many other improvements possible, most mentioned here in comments. I will like to implement them at least in nostr console ( but nothing certain about when that may happen).

Others may also take up and implement these ideas in their/other clients. This is relevant in that this is the simplest way to implement group chat ( with single shared secret, shared with all members), but it has its inherent drawbacks.

[earonesty]

ok, just read that. sending N messages is avoided but you still need to do N encryptions. there's not a big difference really, except now the message is huge. so when someone wants to look for their message, they get a bunch of keys they can't use. i like the encryption in 38 better, yes you have to send more "messages", but they're smaller and this is better for routing, and you can send to different relays, etc.

[earonesty]

#570 (comment)

[earonesty]

other clients won't decrypt it, since it's not kind 4. i think making it "compatible" makes things actually more confusing, not less. better to simply not see it. then you can dm them and be like "bruh get with a new client"

new clients that don't implement chat can chose to show invitations without supporting chat no matter what the format is

and nip04 is kinda broken now (possible iv collisions, structured ss key reuse, etc)

[earonesty]

im building this now, based on the "considered harmful discussions" with @paulmillr mostly as a POC. using the new "versioned encryption" system, ephemeral public invitations and well-known group public/private key, like in this document. working fine so far. you can play with a test flight version soon.

https://github.com/ArcadeLabsInc/arclib/pull/20/files

[staab]

Closing in favor of #574, #686 and possibly #875