FCP HIPE: Wire Messages (formerly AMES) #43

kdenhartog · 2018-09-18T00:23:36Z

This is the JWM work, but it's been renamed to prevent confusion. I'm expecting it to diverge further from the JWE spec some more in order to align with other multiplex designs. It details the format of the serialization and the API that should be used to use this functionality. I expect there to be some discussion and changes in relation to multiplexing described here

rendered text

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

Signed-off-by: Stephen Curran <swcurran@gmail.com>

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

TelegramSam · 2018-09-20T10:59:39Z

AMES/README.md

+
+## Serialization Examples
+
+### JSON Serialization


Is this an example of Authcrypt, or Anoncrypt, or both? How do they differ?

Authcrypt and Anoncrypt only differ 1 but significant way. You're right this should be documented somewhere. I briefly talk about them in this HIPE

Anoncrypt generates an ephemeral key pair, uses the private key with the receivers relationship public key to perform Elliptic Curve Integration Encryption Scheme (ECIES). This means the receiver will not have a cryptographic method to verify who sent it. This offers forward secrecy for the sender. This is used when the sender's public key is not known by the receiver yet but the receivers key is or can be known.

Authcrypt instead of using an ephemeral key pair, uses the senders relationship key pair with the receivers relationship key pair. No forward secrecy is offered. The receiver has a cryptographic method to verify the sender.

I'm splitting the API for this now. I do mention this in the API aspects below, but to make the API cleaner and to stay consistent with the way we anon/authcrypt now I'm planning to separate.

Sorry, I understand AuthCrypt and AnonCrypt. I was looking for how Kyle was demonstrating which.

TelegramSam · 2018-09-20T11:01:19Z

AMES/README.md

+```
+
+### Compact Serialization
+` <header_json> . <content_encryption_key> . <iv> . <ciphertext> . <tag> `


I'm wondering if we adjust the header format to include a list of recipients. That way, multi-recipient messages can be also presented in compact form.

Consolidating to a single serialization format would make the code much cleaner. Both are implemented right now. The format outputted is dependent on the number of keys passed into pack_message at the moment.

TelegramSam · 2018-09-20T11:02:59Z

AMES/README.md

+
+    AMES: This should pass in a json string that follows one of the two serializations provided above.
+
+    my_vk: This should be the verkey that you wish to use to decrypt the message.


Does the message itself contain a clue as to which verkey is needed to decrypt the message? This would be useful (if not a reason against it), and we should either modify the unpack message to not need this argument, or provide a method that would inspect the message and inform about expected keys prior to calling unpack.

I've been under the assumption that this was to be handled by a different layer. If there is a 1 -> 1 mapping of DID -> Key owned by agent it could be removed, but currently there's no way to do this because wallet lookups are dependent on verkeys rather than DIDs.

TelegramSam · 2018-09-20T11:04:41Z

AMES/README.md

+    output: A decrypted message
+
+### pack_message(plaintext, auth, recv_keys, wallet_handle, my_vk) -> JSON String:
+The purpose of this function is to take in a message, encrypt the message for the keys specified and output a JSON string with one of the two serializations identified above.


I think a compact boolean argument to explicitly indicate compact or full encoding is more useful, rather than always going compact for one form and full for the other. If an invalid option is requested, return an error.

I think consolidating to one format only would be a better approach rather than adding another boolean on to determine format. I do agree, that the number of keys determining the serialization format is confusing, but it's what I came up with as I was coding it. Definitely worth a refactor.

TelegramSam · 2018-09-20T11:05:41Z

AMES/README.md

+
+    recv_keys: This is a list of verkeys passed as a string. If only 1 key is passed in, then a Compact serialization will be outputed. If > 1 keys are passed in then JSON serialization will be outputted from this function.
+
+    wallet_handle: this is the wallet_handle that contains the key used to encrypt the message (when using authcrypt). It is required even if using anoncrypt.


Why are both the wallet and the verkey needed here for authcrypt?

So the function knows which wallet to find the private key in. It's not necessary when anoncrypting because an ephemeral key is being generated on the fly.

TelegramSam · 2018-09-20T11:07:29Z

AMES/README.md

+    "recipients" : [
+        {
+            "header" : { 
+                "typ" : "x-b64nacl"


x-b64nacl was intended as a temporary typ value until we defined this format. We should replace this with ames or something to designate this format. Same for alg.

I agree with changing typ. I'm not sure about changing alg. These are intended to show how the cek is being encrypted. I suspect when I do the refactors to encrypt headers (in order to protect to/from keys) then this information will need to be changed as well.

TelegramSam · 2018-09-20T12:17:04Z

AMES/README.md

+### remove_route(did_with_key_frag, wallet_handle):
+This function removes a route (did#key, endpoint) from the routing table for an endpoint based upon the provided parameter. It's expected that this function will not return any data upon completion.
+
+### update_route(DID#key, endpoint, wallet_handle):


This method feels very redundant with add_route. One method that serves as an upsert feels like the right answer.

I preferred separating them to make the code cleaner under the hood. The wallet is handling these two actions as separate as well, so I wanted to continue this pattern. Thanks, for coming in and taking a look at this. There's some good ideas in the suggested refactors.

mikelodder7 · 2018-09-20T16:01:53Z

Here is the rendered text

mikelodder7 · 2018-09-20T16:34:10Z

So how is the entire message protected? I didn't see any method to protect the confidentiality and integrity of the headers? Is this done with authcrypt or anon crypt or using the Signal protocol?

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

kdenhartog · 2018-09-20T18:54:43Z

@mikelodder7 this is a refactor I haven't gone back in and implemented. I expect this to be handled by the function itself similar to how authcrypt handles all of this as well under the hood. This will need to be implemented before it's accepted.

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

TelegramSam

minor fixes.

TelegramSam · 2019-01-07T17:27:41Z

text/AMES/README.md

+        "alg": "Authcrypt",
+        "recipients": [
+            {
+                "encrypted_key": anoncrypt(encrypted_cek|sender_vk|nonce)


should this be anoncrypt(cek|sender_vk|nonce)? the cek isn't yet encrypted (which is the point of this line).

I've changed this to say anoncrypt(authcrypted_cek|sender_vk|nonce). This is implemented this way because this is using crypto_box with a combo_box are both consumed to implement 'authenticated_encrypt' in the libindy/services/crypto/mod.rs file.

TelegramSam · 2019-01-07T17:28:21Z

text/AMES/README.md

+        ],
+    })"
+    "iv": <b64URLencode()>,
+    "ciphertext": <b64URLencode(encrypt({'@type'...}, sym_key)>,


why use sym_key here, when elsewhere it is called cek?

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

kdenhartog · 2019-01-08T19:11:17Z

A message pack dependency crept in from building off the authcrypt code in libindy. Please hold off on approving this HIPE until I've found a way to remove this dependency while remaining in alignment with the libindy work.

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

swcurran · 2019-01-09T15:00:28Z

While this shouldn't hold up moving this HIPE forward (and definitely not the code!), I think there should be more precise examples of the pack() and unpack() interface. What happens in between is important to the code implementers (and is well covered in the HIPE). However, for the users of the capabilities - they really just want to know what they have to pass in to each function, and what is going to be returned.

On that topic - a question I have is about unpack - does it just return the message (like the example in "Code for this might look like the following:" shows), or does it return other necessary information that might be needed routing - notably the verkey? BTW - that example is wrong for the received message - the "tmsg" and "jwe" are both used for the same data element.,

Related question - does the fact that the verkey of the receiver is in the header mean that the "forward" message type is no longer needed? The receiver of a Wire Message could always just get back the verkey of any embedded JWE, and from that, infer (based on routing tables) where the message is to go next. It seems like "Forward" is redundant now that the information is in the JWE.

kdenhartog · 2019-01-09T22:34:06Z

While this shouldn't hold up moving this HIPE forward (and definitely not the code!), I think there should be more precise examples of the pack() and unpack() interface. What happens in between is important to the code implementers (and is well covered in the HIPE). However, for the users of the capabilities - they really just want to know what they have to pass in to each function, and what is going to be returned.

I can add this easily. It's documented in the APIs code, but I'll move it to the HIPE as well, so it's easier for others to implement if they wanted to without looking at the code. Thanks for the suggestion.

On that topic - a question I have is about unpack - does it just return the message (like the example in "Code for this might look like the following:" shows), or does it return other necessary information that might be needed routing - notably the verkey? BTW - that example is wrong for the received message - the "tmsg" and "jwe" are both used for the same data element.,

I'll message you to get the details about the specific example if I can't find it. With regards to the return value, it will return a JSON object that includes a verkey if it's authcrypted. I'll include this in the details that I provide from above.

Related question - does the fact that the verkey of the receiver is in the header mean that the "forward" message type is no longer needed? The receiver of a Wire Message could always just get back the verkey of any embedded JWE, and from that, infer (based on routing tables) where the message is to go next. It seems like "Forward" is redundant now that the information is in the JWE.

I do believe that Forward could be redundant at this point because the verkeys are listed in the recipients header and are publicly viewable. The one advantage that a forward message would have over the verkey's would be the use of DIDs, where as with the wire message we chose not to use DIDs to prevent DID resolution issues from popping up. Instead we think it would be better to build some code on top later that will consume pack/unpack and will handle DID resolution seperately. If done right, forward messages could be removed. Given that the wallet doesn't have a built in verkey -> DID solution right now though, I think it's fine to support forwards without causing future breaking changes.

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

TelegramSam · 2019-01-15T22:05:53Z

text/AMES/README.md

+
+```json
+{
+    "message":"Hello World",


I believe an agent message would be a better example here than a string. Even if abbreviated like {"@type":"" ...}

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

text/0028-wire-message-format/README.md

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

text/0028-wire-message-format/README.md

dhh1128 · 2019-01-23T17:03:10Z

text/0028-wire-message-format/README.md

+# Wire Messages
+[summary]: #summary
+
+There are two layers of messages that combine to enable **interoperable** self-sovereign identity Agent-to-Agent communication. At the highest level are Agent Messages - messages sent between Identities (sender, recievers) to accomplish some shared goal. For example, establishing a connection between identities, issuing a Verifiable Credential from an Issuer to a Holder or even the simple delivery of a text Instant Message from one person to another. Agent Messages are delivered via the second, lower layer of messaging - Wire. A Wire Message is a wrapper (envelope) around an Agent Message to permit sending the message from one Agent directly to another Agent. An Agent Message going from its Sender to its Receiver may be passed through a number of Agents, and a Wire Message is used for each hop of the journey.


Would be good to link here to HIPE 0026-agent-file-format. Note that that HIPE uses the term "Agent Plaintext Message" and "Agent Wire Message"; it does not use "Agent" for one and "Wire" as the contrasting word. We could adjust the text here by saying "At the highest level are agent plaintext messages--messages sent...". Then later we could say, "Agent plaintext messages are delivered via the second..." and "An agent plaintext message going from its sender to its receiver may be passed through a number of agents, and a wire message envelope is used for each hop of the journey."

text/0028-wire-message-format/README.md

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

Signed-off-by: Daniel Hardman <daniel.hardman@gmail.com>

Explain for non-Indy audience

dhh1128 · 2019-01-25T23:26:20Z

HIPE has been at FCP for quite a while. Implementation was merged into Indy SDK today.

lovesh · 2019-03-14T09:23:55Z

text/0028-wire-message-format/README.md

+```json
+{
+    "protected": "eyJlbmMiOiJ4Y2hhY2hhMjBwb2x5MTMwNV9pZXRmIiwidHlwIjoiSldNLzEuMCIsImFsZyI6IkF1dGhjcnlwdCIsInJlY2lwaWVudHMiOlt7ImVuY3J5cHRlZF9rZXkiOiJMNVhEaEgxNVBtX3ZIeFNlcmFZOGVPVEc2UmZjRTJOUTNFVGVWQy03RWlEWnl6cFJKZDhGVzBhNnFlNEpmdUF6IiwiaGVhZGVyIjp7ImtpZCI6IkdKMVN6b1d6YXZRWWZOTDlYa2FKZHJRZWpmenRONFhxZHNpVjRjdDNMWEtMIiwiaXYiOiJhOEltaW5zdFhIaTU0X0otSmU1SVdsT2NOZ1N3RDlUQiIsInNlbmRlciI6ImZ0aW13aWlZUkc3clJRYlhnSjEzQzVhVEVRSXJzV0RJX2JzeERxaVdiVGxWU0tQbXc2NDE4dnozSG1NbGVsTThBdVNpS2xhTENtUkRJNHNERlNnWkljQVZYbzEzNFY4bzhsRm9WMUJkREk3ZmRLT1p6ckticUNpeEtKaz0ifX0seyJlbmNyeXB0ZWRfa2V5IjoiZUFNaUQ2R0RtT3R6UkVoSS1UVjA1X1JoaXBweThqd09BdTVELTJJZFZPSmdJOC1ON1FOU3VsWXlDb1dpRTE2WSIsImhlYWRlciI6eyJraWQiOiJIS1RBaVlNOGNFMmtLQzlLYU5NWkxZajRHUzh1V0NZTUJ4UDJpMVk5Mnp1bSIsIml2IjoiRDR0TnRIZDJyczY1RUdfQTRHQi1vMC05QmdMeERNZkgiLCJzZW5kZXIiOiJzSjdwaXU0VUR1TF9vMnBYYi1KX0pBcHhzYUZyeGlUbWdwWmpsdFdqWUZUVWlyNGI4TVdtRGR0enAwT25UZUhMSzltRnJoSDRHVkExd1Z0bm9rVUtvZ0NkTldIc2NhclFzY1FDUlBaREtyVzZib2Z0d0g4X0VZR1RMMFE9In19XX0=",
+    "iv": "ZqOrBZiA-RdFMhy2",


Any reason why iv not called nonce as called in libsodium. I see indy-sdk code does not call it nonce either, is that the reason or something else?

lovesh · 2019-03-14T09:32:06Z

text/0028-wire-message-format/README.md

+The protected data base64URL decodes to this:
+```json
+{
+    "enc": "xchacha20poly1305_ietf",


The code we have in indy-sdk seems to be using xsalsa20 and not xchacha20? Is there a proposed change to xchacha20? Or does this enc refer to encryption method of the message and the method to encrypt will allways remain libsodium's crypto_box?

FCP HIPE: Wire Messages (formerly AMES) Signed-off-by: Brent <brent.zundel@gmail.com>

kdenhartog and others added 14 commits July 12, 2018 20:23

added description of AnonCrypt and AuthCrypt

659d3cb

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

reversed sender and random priv_key in step 3

af5e21e

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

Merge branch 'master' of github.com:hyperledger/indy-hipe

1018805

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

Initial version of Wire Message HIPE

345c5c1

Signed-off-by: Stephen Curran <swcurran@gmail.com>

Clarifying the requirements of the unpack() when using the authcrypt

876092b

Signed-off-by: Stephen Curran <swcurran@gmail.com>

Args to pack() must include myPubKey when AuthCrypt being used

8fcce07

Signed-off-by: Stephen Curran <swcurran@gmail.com>

Merge branch 'master' of github.com:swcurran/indy-hipe

d5e5001

Merge branch 'wireMessages' of github.com:swcurran/indy-hipe

f23d11e

fixed prior art sections to match implementation details

15baaca

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

Merge branch 'master' of github.com:hyperledger/indy-hipe

4178bcc

AMES (aka JWMs) HIPE

1b7f276

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

updated summary

6942efa

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

fixed location and removed non-accepted hipes

1c39f47

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

updated unpack API

32681b5

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

TelegramSam reviewed Sep 20, 2018

View reviewed changes

kdenhartog added 4 commits September 20, 2018 12:07

separated Route Table work - will go in separate HIPE

3035b33

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

Removed route table docs, will move to separate hipe

a2d5214

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

moved hipe to text directory only

db5de9a

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

updated motivations; removed other mention of route tables

08ed97d

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

kdenhartog added 2 commits September 20, 2018 13:12

Added link to code to track progress

36a6dcf

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

updated HIPE to fit API format

1255ed8

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

TelegramSam requested changes Jan 7, 2019

View reviewed changes

kdenhartog and others added 3 commits January 7, 2019 11:24

tweaked to clarify in response to telegramsam comments

fd3d8cc

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

Merge branch 'AMES' of github.com:kdenhartog/indy-hipe into AMES

d777fb2

Merge branch 'master' into AMES

c955e3a

TelegramSam approved these changes Jan 7, 2019

View reviewed changes

kdenhartog added 2 commits January 8, 2019 15:10

added examples of json

0b09c02

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

Merge branch 'AMES' of github.com:kdenhartog/indy-hipe into AMES

06739e8

added additional details to reflect @swcurran comments

1db7d6e

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

TelegramSam reviewed Jan 15, 2019

View reviewed changes

aligned HIPE with final implementation

ebc2339

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

dbluhm reviewed Jan 22, 2019

View reviewed changes

text/0028-wire-message-format/README.md Outdated Show resolved Hide resolved

kdenhartog added 4 commits January 22, 2019 22:35

added recipient_verkey

8e7dfd2

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

updated unpack algorithm details

9d135e2

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

fix typo

cc5fdb6

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

updated docs

f586b31

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

dhh1128 suggested changes Jan 23, 2019

View reviewed changes

text/0028-wire-message-format/README.md Outdated Show resolved Hide resolved

kdenhartog and others added 3 commits January 23, 2019 14:57

added changes based on new comments

bb55f79

Signed-off-by: Kyle Den Hartog <kdenhar@gmail.com>

Explain for non-Indy audience

da6c1fc

Signed-off-by: Daniel Hardman <daniel.hardman@gmail.com>

Merge pull request #3 from dhh1128/ames

36a92cc

Explain for non-Indy audience

dhh1128 approved these changes Jan 25, 2019

View reviewed changes

dhh1128 merged commit 82629c3 into hyperledger:master Jan 25, 2019

lovesh reviewed Mar 14, 2019

View reviewed changes

brentzundel pushed a commit to brentzundel/indy-hipe that referenced this pull request Apr 16, 2019

Merge pull request hyperledger#43 from kdenhartog/AMES

a155b45

FCP HIPE: Wire Messages (formerly AMES) Signed-off-by: Brent <brent.zundel@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FCP HIPE: Wire Messages (formerly AMES) #43

FCP HIPE: Wire Messages (formerly AMES) #43

kdenhartog commented Sep 18, 2018 •

edited

Loading

TelegramSam Sep 20, 2018

mikelodder7 Sep 20, 2018

kdenhartog Sep 20, 2018

TelegramSam Sep 25, 2018

TelegramSam Sep 20, 2018

kdenhartog Sep 20, 2018

TelegramSam Sep 20, 2018

kdenhartog Sep 20, 2018

TelegramSam Sep 20, 2018

kdenhartog Sep 20, 2018

TelegramSam Sep 20, 2018

kdenhartog Sep 20, 2018

TelegramSam Sep 20, 2018

kdenhartog Sep 20, 2018

TelegramSam Sep 20, 2018

kdenhartog Sep 20, 2018

mikelodder7 commented Sep 20, 2018

mikelodder7 commented Sep 20, 2018

kdenhartog commented Sep 20, 2018

TelegramSam left a comment

TelegramSam Jan 7, 2019

kdenhartog Jan 7, 2019

TelegramSam Jan 7, 2019

kdenhartog commented Jan 8, 2019

swcurran commented Jan 9, 2019

kdenhartog commented Jan 9, 2019 •

edited

Loading

TelegramSam Jan 15, 2019

dhh1128 Jan 23, 2019

dhh1128 commented Jan 25, 2019

lovesh Mar 14, 2019

lovesh Mar 14, 2019 •

edited

Loading


		AMES: This should pass in a json string that follows one of the two serializations provided above.

		my_vk: This should be the verkey that you wish to use to decrypt the message.


		recv_keys: This is a list of verkeys passed as a string. If only 1 key is passed in, then a Compact serialization will be outputed. If > 1 keys are passed in then JSON serialization will be outputted from this function.

		wallet_handle: this is the wallet_handle that contains the key used to encrypt the message (when using authcrypt). It is required even if using anoncrypt.

FCP HIPE: Wire Messages (formerly AMES) #43

FCP HIPE: Wire Messages (formerly AMES) #43

Conversation

kdenhartog commented Sep 18, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikelodder7 commented Sep 20, 2018

mikelodder7 commented Sep 20, 2018

kdenhartog commented Sep 20, 2018

TelegramSam left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kdenhartog commented Jan 8, 2019

swcurran commented Jan 9, 2019

kdenhartog commented Jan 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhh1128 commented Jan 25, 2019

Choose a reason for hiding this comment

lovesh Mar 14, 2019 • edited Loading

Choose a reason for hiding this comment

kdenhartog commented Sep 18, 2018 •

edited

Loading

kdenhartog commented Jan 9, 2019 •

edited

Loading

lovesh Mar 14, 2019 •

edited

Loading