NIP-44: Encrypted Direct Message (Versioned), replaces NIP-4

[paulmillr]

Simple solution that fixes broken cryptography and introduces versioning. Preview, PR to nostr-tools

tl;dr

• confidentiality is achieved by xchacha(key: sha256(ecdh(userPriv, partnerPub)), nonce: random(24)).encrypt(message)
• integrity is achieved by ciphertext getting signed with user's secp256k1 schnorr key
• versioning for future-proofness
• profiles signal which relays should be used to communicate with them
• no one could fetch DM events besides "the authenticated user"

Benchmark

Round-trip of encrypting-decrypting a basic event with "hello" message, on Apple M2.

nip4 x 19,419 ops/sec @ 51μs/op
nip44 x 101,235 ops/sec @ 9μs/op

The goals

1. Security
2. Quick and dirty first version, without advanced features. We can iterate further after the adoption.
3. Simplicity. We need to get maximum adoption by nostr client and relay implementors. NIPs are optional and no one is forcing anyone to implement them. If we overcomplicate things, no one would write code for it.
4. As few building blocks as possible. Implementors should not be able to shoot themselves into the foot: "dumb cryptography". Users should not be able to leak their nostr private key.

NIP4 was bad, because it used AES-CBC with non-hashed, non-IND ECDH key, was not versioned, did not have relay signalling.

Because it's simple, the protocol would not be as good as Signal or others. It's okay, because nostr is public DM system. Ideally you'll exchange Signal (etc) credentials in nostr DMs. If you think the approach is wrong, write down your thoughts in this thread.

Rationale

• One static shared key is used to encrypt all messages between person A and person B
  • We could have rotated keys: encrypting every message with a new one. The key could be generated from some kind of salt, that would be attached to message
    • If user key is leaked, the attacker would be able to decrypt the message anyway. Complication for nothing
• encrypt-then-sign, XChaCha20 is extended-nonce construction of ChaCha20, RFC8439-standardized algorithm
  • It allows us to use same encryption key for all messages. The safe threshold of key wear-out for XChaCha is up to 2^80 messages to be sent under a single key. For AES it's just 2^32!
  • It is simpler, allowing us to use 24-byte nonces and same encryption key for all messages. AES-GCM uses 12-byte nonces which are too short to pick randomly, because of collision risk. Which means a separate key needs to be used per-message, which means we'll need to bring KDF, and complicate the scheme
  • It is faster than AES on devices which don't have hardware acceleration: those include many cheap phones.
  • It is also much faster (14x) for our precise case than AES because there is no KDF step
• No AEAD (Authenticated encryption) aka Poly1305
  • Any tampering, bit flipping will invalidate the ciphertext's signature
  • Poly1305 is not perfect, its integrity limit for single-key messages is 2^46
  • Polynomial, non-committing MACs are vulnerable to invisible salamanders attack
• SHA256 is used to hash shared secret instead of HKDF
  • sha256 is already standardized for this purpose in libsecp256k1
  • Achieves key IND 
  • HKDF is easy to misuse (salt? info?)
  • HKDF is 10x slower than sha256
  • HKDF introduces a new algorithm into the lightweight nostr protocol, which means more code
  • HKDF is awesome if we were to construct a new version with forward secrecy, but not for now
• Only one encryption algorithm for now, instead of a choice between xchacha and aes. Makes it mandatory for clients to implement without diverging

Out-of-scope

• Ephemeral anonymous keys. Prototyped in NIP-444 and NIP-59 gift wrapping
• Forward secrecy. It's great feature: when a user leaks their key, adversary won't be able to read their previous messages. But it also means users themselves won't be able to read their previous messages. nostr clients need to be able to fetch all historical conversations for UX purposes; nostr clients are immature and it's more-or-less common for users to switch between them. We can get back to forward secrecy prototyping in the future algorithm versions.
• Really private and secure messaging. nostr will always leak metadata. It's better to keep this in mind and only use nostr for exchanging contacts for more secure messenger

Questions

• Should content be hex string, or CSV base64 string?
• Is AEAD really not needed?
• Should we just prepend 02 to B in getSharedSecret(privkeyA, '02' + pubkeyB)?
• How could this work with group chats? Signal group chats are simply a bunch 1-to-1 conversations. Maybe we could think of something similar here

Live code: #574

[fiatjaf]

What if this NIP also included:

• AUTH (NIP-42) requirement for relays to return chat messages (even if relays don't implement that at first, clients should be ready for when they do).
• Explicit relay signaling: users must signal via NIP-65 that they want to receive DMs only on some specific relays, and whenever one wants to message these users they should only always send DMs to these relays and only those.

I think these two small measures will reduce metadata leak to basically zero in normal times and also reduce bandwidth significantly.

[paulmillr]

@fiatjaf sounds good. How does the new thing look like in general to you? And WDYT about #468?

[earonesty]

minor comments, looks great

[starbackr-dev]

AUTH (NIP-42) requirement for relays to return chat messages (even if relays don't implement that at first, clients should be ready for when they do).

if we make AUTH (NIP-42) mandatory then we can drop NIP-59 #468 completely because there is no use for it.

[earonesty]

making AUTH mandatory is a nonstarter, imo. also the disclaimers in this nip are misleading. if metadata is not a concern (and there's no reason we should blanket assume that it is), there is no need to use AUTH. i can think of many good reasons why you definitely want metadata to be present in this shared-custody data

also, there are other solutions ... it would be easy to gift-wrap (nip59?) or to have a system send spurious messages regularly to preventing harvesting of this info.

the enryption provided in this leaks the sender and the recipient public keys. that's the only disclaimer we need. it is correctly done now and scaring people away from using it isn't the right move.

[starbackr-dev]

We got to pick one option or the other to stop metadata leaks with this.

Option 1: Make NIP-42 mandatory and explicit relay signaling in the profile.
or
Option 2: Encrypt metadata as per NIP-59 but free to post on any relay.

Option 2 wastes compute and bandwidth resources but less restrictive.

But we have to do either one and close this topic for a while.

one more. I am new to this NIP process. Shouldn't @earonesty name be on the NIP-44 since he started this discussion and did initial work?

[paulmillr]

@starbackr-dev

he started this discussion

The initial discussion was started by mikedilger in #72, adiabat in #107 (w me proposing chacha on feb 11), AdamISZ in #303. I also don't know how nip authoring process works. Does anyone know?

[earonesty]

i don't care, in the slightest, if my name is on it. i understand the tradeoffs and i think good decisions are being made, and it basically aligns with what i need. i think metadata "leakage" (if it can be called that), is outside the purview of this nip, and it should just be made clear that sender public key info is visible and that additional tags should not be used on these messages.

1. very easy to hide the sender by rolling new pubkeys (see nip 59 which i love)
2. very easy to authenticate with relays using a mechanism that isn't your public key (probably a bad idea anyway to use that)

metadata is not "leaked" ... this is a conversation that the sender can decrypt later, because nostr is an censorship resistant data store, and users want to be able to move between clients and see their outbounds too.

(technically recipient information doesn't matter and cannot matter)

[mikedilger]

I like this NIP. I think it makes the right decisions. We can't do forward-secrecy or metadata secrecy in nostr without a lot of other junk (if at all) and we need a better but still simple DM system.

Explicit relay signaling: users must signal via NIP-65 that they want to receive DMs only on some specific relays, and whenever one wants to message these users they should only always send DMs to these relays and only those.

I like this. I could specify my personal relay that doesn't allow anybody to read back what was written.

The initial discussion was started by mikedilger in #72, ...

Don't put me as an author. I didn't think you were going to, but just for clarity.

[earonesty]

NIP-59 handle metadata secrecy elegantly and very well. It does exactly what nostr should do. We just need to update NIP-59 to specify that it uses NIP-44 encryption instead of NIP-04 and now we've got the building blocks of a a lot of great protocols!

[BlowaterNostr]

@paulmillr Do you have an working example of the padding oracle attack?

[paulmillr]

@BlowaterNostr no

[ghost]

Question, how does this perform against on-the-fly modifications of the public key, signature and content?

If Mallory would change Alice's message, right before it reaches the relay, and changes the public key so it matches hers, changes the ciphertext to something else and recomputes the signature, is there any way to validate that the message actually came from Alice?

Why not pass that data into Poly1305 as AAD, so it's protected from being modified?
Or am I wrong?

[paulmillr]

@iloveyumyumshrimpnoodles how would poly1305 protect anything in this case? mallory's new ciphertext would include new poly authentication tag as well. It would validate properly.

[earonesty]

at this point Mallory has changed the entire message to something she felt like sending

it is just a message from Mallory now

she might as well just posted it from the outset

If she's changing the content and the public key and the event hash and everything

[ghost]

Ah yeah, true.
My bad, haven't got much sleep

[earonesty]

simple feature. instead of signing with his public key, a client signs with (shared-secret*privatekey) instead. then the recipient of the dm can verify the message using (pubkey*sharedsecret), but nobody knows who sent it, and also the recipient can't prove who sent it without releasing his private key. clients can calculate an authors: filter easily as well, this gets a lot of privacy with little effort, and it can be marked OPTIONAL, because the only downside is a lack of validation (clients will still show the messages)

[paulmillr]

Stealth addresses are hard and require complicated analysis to ensure discrete logarithm problem is not stupidly solvable. i've been around an audit of one such system. Better to not have it in the initial release.

[earonesty]

Stealth addresses are hard and require complicated analysis to ensure discrete logarithm problem is not stupidly solvable. i've been around an audit of one such system. Better to not have it in the initial release.

i would hope that signing with a tweaked private key even if everyone knows how you tweaked it, cant possibly be broken or basically all derivations are broken

[fiatjaf]

Sorry, @paulmillr, I forgot to answer #574 (comment).

I think "gift wrap" and all the other similar proposals that use random keys or anonymous keys are bad because for the incentives to work on Nostr relays ultimately will have to know something about who is writing to their database, so these solutions won't scale or may require additional complexities (despite already being complex and resource-intensive for clients).

[fiatjaf]

How do you feel about including #574 (comment) in the NIP?

[paulmillr]

1. I've added NIP-42 and NIP-65. cc @fiatjaf. However, one case is unclear to me. What if I want to communicate with Alice, via specific, "more secure" relay that we both agreed upon, which is not in the list of our preferred relays? Also, how does this play with wrapper "hidden" messages that have been mentioned above?

2. With regards to content encoding, our options are:

   • hex, with splits at specific indexes ((0, 2), (2, 34)). Pros: consistency with sig and pubkey fields 1c35e24e540566e8464dfe2d5c43453d936207fdfe
   • base64+csv. It is 1.5x shorter than hex and can save some traffic. Disadvantages would include the need to include base64 implementation (currently no other NIP requires it, besides NIP4, which we're replacing). 1,hkMpXxHhTPZA3gFYrsJPWEfGO6YBc0Lw,5EZtlb1tF4uf

3. With regards to algorithm choice:

   • iOS has ChaCha20-Poly1305 https://developer.apple.com/documentation/cryptokit/chachapoly
   • Android has ChaCha20, ChaCha20-Poly1305 https://developer.android.com/reference/kotlin/javax/crypto/Cipher
   • HPKE rfc9180 has ChaCha20-Poly1305 https://datatracker.ietf.org/doc/rfc9180/
   • MLS has ChaCha20-Poly1305 https://datatracker.ietf.org/doc/draft-ietf-mls-protocol/
   • Sodium has box/sealedbox/secretbox xsalsa20-poly1305, secretstream xchacha20-poly1305
   • tweetnacl has xsalsa20-poly1305

Before merge, I also want to have a second opinion from someone experienced, who can do security evaluation / audit. If anyone knows such people, it would massively help.

[BlowaterNostr]

Is metadata leakage and cryptographic security 2 several problems? Can we solve them independently?

Of course, some good designs might solve them at once, but it's not a good mentality to have to solve them at once. If we can solve one without making the other worse, client developers will catch up faster.

Recently I don't even have time to carefully evaluate this NIP despite Blowater is mainly focusing on DM.

[fiatjaf]

Thinking again about this, I think people won't switch easily, so it may be better to try to implement SimpleX instead? Even though it is too complicated, someone can take an OpenSats grant and write a SimpleX JS library and libraries for other languages, lobby the SimpleX people to enable websocket support on their servers or write a separate implementation, and allow easily hosting paid SimpleX relays or community whitelisted SimpleX relays, then we have a solid feature to build upon (assuming SimpleX is actually good).

[paulmillr]

I strongly suggest we finalize NIP-44, explicitly state our threat model, do some sort of audit of the result scheme, and release it to public. This is because Simplex is extremely complicated. I will open a separate discussion with regards to Simplex estimates and amount of work, it's likely 100x more than finalizing NIP-44.

Regarding "people won't switch easily" - on a contrary, I don't think so. We just need to sell it. If it's too complex, then yes, they won't switch. If it's as simple as it gets, then switching should take no time. It's important to understand that, in the worst case, all the primitives (xchacha, whatever) are still available in extremely popular libsodium, which works on all platforms. So, it's good. For JS i've released noble-ciphers, so it's good too.

To reiterate deficiencies of NIP4: 1) cbc padding oracle attacks 2) multi-user security of AES is shit 3) cross-platform key reuse, it's easy to accidentally leak private key 4) and aes key not being IND. Maybe 4 is minor. 1 and 2 are medium. 3 is really bad, since nostr uses one key for everything.

@BlowaterNostr metadata leakage of some sort always happens. It happens in signal, it happens in tor, it happens everywhere. The best we can do is minimize it. I think fiatjaf's solution would radically decrease leakage, so it's cool.

[paulmillr]

SimpleX discussion => #658

[earonesty]

we're using the sha256(shared secret) to sign/encrypt messages. but are otherwise using nip-44 for encryption. that prevents metadata leakage better than most of the above. and its plausibly deniable. and it's understandable/easy to use. i really would like something in nip-44 that mentions that this is an easy option to add "metadata free dm's" to any nostr client

[paulmillr]

@earonesty

and its plausibly deniable

how so?

[mikedilger]

How about mentioning that these are Alice's read relays, the ones she reads from (in case she uses relays for just read or just write, instead of both).

... or better yet, since people are concerned about metadata, we extend NIP-65 to add a flag indicating a DM relay. That way it would be possible for people concerned with metadata leakage to setup a personal relay for receiving DMs and be confident that those DMs are not being read by interlopers (at least the receipient is confident it doesn't leak from their specified read relay, but not that the sender didn't put it elsewhere; and the sender can't really be confident of this, so this is not much of a solution for metadata leakage but it helps a smidgen).

[vitorpamplona]

Folks, I can't find a reliable xchacha20 implementation for Android. Do you have any tips? It's going to be hard to support this PR if the cryptographic function is not easily available in all languages. :(

[paulmillr]

@vitorpamplona libsodium is available on all platforms.

[paulmillr]

To clarify, in libsodium the function is called crypto_stream_xchacha20. There are bindings to basically all languages, that includes java, js, swift, and everything else.

[vitorpamplona]

I got it to work with one of the bindings: #468 (comment)

3/4 of the libs I tested were crashing for no reason. Things are still very raw.

[earonesty]

we're using a variant of this in arcade, running on android. it doesn't use chacha, but it does the "right thing" for aes. (new salt, new key per message): https://github.com/ArcadeLabsInc/arclib/blob/9857e1ae1558b9e265c1461a53d281ecbe4fae77/src/ident.ts#L107

works fine in react/ios/andriod and is def better than nip04. also we support blinded-dms as well (no metadata leaks) on android

i had various problems using GCM and chacha, so this is where i left it

[paulmillr]

your solution is still:

• much less secure against multi-key attacks
• vulnerable to padding oracle attacks
• not compatible with nip4. if an upgrade is required, it should be a proper upgrade

i had various problems using chacha

Which problems, exactly?

[shocknet-justin]

Dropping an idea I had around metadata concerns that would not alter this NIP as proposed, because if this is more cryptographically sound than NIP04, it should move forward on that basis alone.

Metadata can be addressed in a separate NIP, using this one as a dependency to keep those concerns separated.

"Chaum Chat"

The objection to ephemeral conversation keys ties back to paid relays not allowing writes for new keys. With all he stupidity being pushed by NGO's to use ECash in representing custodial Bitcoin, why not re-purpose those libraries into something useful, like relay access tokens?

An additive NIP for paid relays could define how clients receive ECash instead of a whitelisting, with the ECash then being used for the whitelisting of keys.

Additionally, free relays could reward ECash to identity keys based on reputation, to keep rate limits loose without abuse.

This process would decouple identity keys and their ephemeral children from the Lightning payment.

Since's it's only for service access, there's no dishonest NGO narrative of it enhancing the security/privacy of Bitcoin transactions themselves. It also serves as a demonstration of how ECash should be used.

[vitorpamplona]

"Chaum Chat"

I can't evaluate the merits of this idea without a full description. But I am happy to do so if you decide to make a NIP for it. It feels like Large Group chats (~1,000+ people) could use this Ecash allowlist idea.

[xz-cn]

Correct me if I am wrong, but I have a question regarding this NIP as well as NIP-04. So the ECDH part involves computation using the private key. Where exactly should this computation getConversationKey take place?

Ideally, users should be able to store their private key in an extension wallet such as Metamask of Ethereum. But Metamask won't let a Dapp export your private key for an ECDH computation. For Nostr, idealy it should be same—web app can't touch your private key directly. Maybe we should have a Nostr wallet which does signature and ECDH computation within the wallet instead of exporting it to a web app. Thoughts?

[paulmillr]

Of course. That was one of ideas behind getConversationKey. It can be abstracted in a proper way.

[staab]

I've created a new version of this PR at #715, @paulmillr please let me know if you'd like to close this PR or incorporate those changes into this one.

[staab]

Closing this since the conversation is continuing at #715