Draft EIP: Whisper packet codes spec

[gluk256]

No description provided.

[Arachnid]

I think this is a good start, but needs fleshing out a lot more if it's to be useful to other implementers.

[Arachnid]

You should probably put these in a table with a column for EIP, so later EIPs can extend this set.

[gluk256]

where can i see an example?

[Arachnid]

Which are defined where?

[gluk256]

at the moment they are defined only in the source code of v5 implementation

[Arachnid]

s/whsiper/whisper/

[Arachnid]

s/decrypted/decoded/

[Arachnid]

Should messages be forwarded even if they can't be decoded? What if they contain forwarding rules that the node can't understand?

[gluk256]

All messages are sent to everybody else, if they are valid (good format, not expired, good PoW, etc.). Decryption is only possible for the node that holds the corresponding key.

[Arachnid]

Are all messages always broadcast everywhere? I was under the impression that some messages could be point-to-point, depending on the payload.

[Arachnid]

Why have both an expiry and a TTL?

[gluk256]

To make sure that message is not created in the future.

[Arachnid]

I don't follow; could you not just set an expiry time, and ditch the TTL?

[rphmeier]

actually a decent point. what's important to the receiver is how long the message has left to live, not the time that it was initially meant to be live for.

[Arachnid]

Does the length prefix include the length bytes? How do I know how many length bytes to read?

[gluk256]

Does the length prefix include the length bytes? -- fixed.
How do I know how many length bytes to read? -- explained in the next line.

[rphmeier]

Mail server and relay of non-live messages should happen at a layer above the message relaying protocol or within a separate subprotocol. I vote to keep whisper as simple as possible by including only messages relevant to live envelope propagation.

[gluk256]

It is not only mail. There might be some other cases when you can not afford to spend time on PoW, e.g. some kind of real-time data feeds, etc.

[rphmeier]

Those are valid use-cases, and that's fine, but in the aim of keeping the whisper protocol simple it is better to make this an opt-in subprotocol or happen via some other mode of communication with the trusted peer.

[gluk256]

i disagree with the notion that adding an extra message type suddenly makes whisper complicated. especially because this type is optional. if you don't implement it -- fine, just gracefully ignore this message. actually, this is required behavior for any unknown type anyway. the reason that this type is in the EIP is because it is reserved for this purpose. but it it not mandatory.

[rphmeier]

Should be 64-bits for future-proofing

[gluk256]

can you elaborate? what is your concern?

[rphmeier]

After 2038 this type will be too small to hold future timestamps. I think it is better to future-proof it from the beginning without requiring a protocol upgrade at some point.

[rphmeier]

I disagree with the inclusion of AESNonce -- the relaying of whisper envelopes should be completely generic over the contents of the payload. As long as the application using it (e.g. RPC API) expects a certain format to payloads with specific topic, we can include such metadata in there.

[gluk256]

Whisper can not be encryption-agnostic because it implements the filters, decrypts messages and delivers only decrypted messages to the Dapps.

[rphmeier]

If it expects that messages with matching filters will contain payloads encrypted with a certain key/scheme pair (which is likely, given that it's a probabilistic indicator), then I don't see why it wouldn't work. Topic collision to this degree is already possible even without making it encryption-agnostic.

[rphmeier]

My previous comments about encryption make version number unnecessary.

[gluk256]

please see my comment above

[rphmeier]

This can be done on another subprotocol

[rphmeier]

At the level of the core wire protocol, I think the only concerns should be

1. is the message well-formed (w.r.t. the header, not the payload)?
2. is the message expired?
3. does the message have sufficient PoW for my local requirement?

We can have additional messages to peers detailing things like PoW or topic requirements in order to restrict the incoming set.

All the described checks for payload validity, encryption, and so forth can be done at the application level. This also leaves us open to using the whisper relay network for a larger variety of applications, including those which broadcast plaintext or use alternate encryption. A less complicated message format means doing less work for packets that don't match your local filter.

I would expect the RPC layer to handle things like standard message format, extracting a metadata such as AES nonce, encryption, signing, and decryption. Having a more general wire protocol lets us upgrade these standards without diminishing the connectivity of the network. The version number mandated here seems to me like a workaround to the same problem, but I think it's a worse solution.

[rphmeier]

what does this floating point value indicate? PoW and PoW heuristic calculation should be specified, otherwise nobody will come to agreement over what the "requirement" means.

Here's a decent method:

fn short_rlp(envelope) = rlp of envelope, excluding env_nonce field.
fn pow_hash(envelope, env_nonce) = sha3(short_rlp(envelope) ++ env_nonce)

lower pow_hash is better (treating as big-endian 32-byte integer)

To calculate PoW heuristic:

fn work_to_space_time_ratio(pow_hash, size, ttl) 
    = 2**leading_zeros(pow_hash) / (size * ttl)

where size is the size of the full RLP-encoded envelope.

[gluk256]

thanks! this is how it is implemented, i will update the spec accordingly.

[rphmeier]

perhaps an even better metric would be to split the hash into two bit vectors: leading_zeros and nonzero_part and then define

fn work_to_space_time_ratio(pow_hash, size, ttl)
    = 
        let partial_len = min(len(leading_zeros), len(nonzero_part));
        (2**len(leading_zeros) + (nonzero_part[0..partial_len] as uint) / (size * ttl)

to avoid work ramp-up being purely exponential. Although then you would likely get more floating point drift in the division.

[rphmeier]

bloom function needs to be specified to define protocol misbehavior

[gluk256]

good point, i will do it

[rphmeier]

Why not a list of topics? Issuer pays for the PoW of more topics anyway.

[gluk256]

Because the purpose of Topic is probabilistic indication of encryption key (should we try to decrypt the message?), and not to indicate anything about the content of the message. Every message can be encrypted with only a single key, hence single Topic.

[rphmeier]

You can have other topics though. For something like a broadcast you may multicast to multiple topics rather than only to a single one. This opens up the ability to use one-time keys for convenient broadcast with a pretty simple scheme (using a chat room as a concrete example):

• Encrypt the message with a 32-byte one-time use key K (and globally agreed-upon nonce). Call the ciphertext C
• Topics: [sha3²(chat_room_name_1), sha3²(chat_room_name_2), ...]
• Payload: [sha3(chat_room_name_1) ^ K, sha3(chat_room_name_2) ^ K, ..., C]

The payload length is always (32 * n_topics) + len(C).
Now anybody who knows any one (not all!) of the chat room names can decrypt the messages by
a) checking for the topic
b) extracting the one-time key
c) decrypting the plaintext

[gluk256]

This is very narrow case, which can be done on Dapp level without further complicating the Whisper protocol. E.g. every time somebody is invited to join a chat room, he receives a single message with all the details (session key, possible topics of outgoing messages, possible topics of incoming messages, etc.).

[rphmeier]

It is just one of many use cases made possible by generalizing Whisper over encryption type and giving the ability to provide many topics. Can you find a way to multicast as efficiently with the spec here?

[kaikun213]

Would agree, multiple topics would enable multicasting with much less work done. This is an advantage in a lot of usecases.

[rphmeier]

Relaying only a single message per packet? It should be more efficient to allow batching.

Previous version of whisper also used the packet as a form of keep-alive: both peers on a connection would take turns sending (potentially empty) Messages packets, starting with the peer with lower node key (comparing big-endian) and always responding within 1 second.

[gluk256]

it is better to leave it to the Dapp level.

[rphmeier]

For using it as keep-alive, that's fair. But relaying multiple messages from your pool at a time is strictly an efficiency concern and can only be addressed at the wire protocol level.

[rphmeier]

I like the addition of padding, but as per my other comments it is not the concern of the wire protocol

[gluk256]

If we leave it to the Dapp level, very few people will implement it, and those who do will look very suspicious. This is the whole point of plausible deniability -- it should be implemented by default and used by everyone, especially by those who don't actually need it.

[rphmeier]

Good point. It still can be done at the RPC level.

[rphmeier]

agreed on data field format, but I think it actually belongs in the upcoming whisper RPCs EIP under a "Common Whisper Payload Format" section

[gluk256]

As RPC EIP does not exist yet, we should at least mention the payload format here. Maybe I should say it is temporarily here?

[rphmeier]

that makes sense to me. i can start to draft the RPCs EIP shortly as well

[gluk256]

fixed

[rphmeier]

wouldn't it be n += 256, since it's meant to be treated as the 9th bit? adding 512 would overflow, but also be congruent mod 512 in the first place.

[gluk256]

indeed, thanks!

[rphmeier]

main problem here is that without Expiry, TTL, and Nonce, we can't know when to evict a "future version" message from the pool. I would say it should be illegal to forward a message to a peer which isn't decodable by them, according to their whisper version.

[rphmeier]

so new version nodes can continue to forward old version formats, but old version nodes can't forward new version formats. nodes should also have some idea of which version numbers actually change the format

[rphmeier]

also: clarify what to set version to.
is it always equal to the protocol version of the creating node? or equal to an alternate envelope format version? not all protocol version upgrades will change the envelope format.

[gluk256]

i disagree. all envelopes should be forwarded further, unless envelope is invalid. besides, when you start a couple of nodes with a new version they will not be able to send any messages because most likely they will be surrounded by the nodes with the old version.

[gluk256]

the purpose of version is to indicate the encryption algorithm, in case we will add a new one, or we will have multiple clients using different algorithms (e.g. geth vs. parity).

[rphmeier]

hm, but since payload is general then version should be contained within there, rather than within the envelope. I was under the impression that version here was to signify the envelope format version (if extra fields were added or something), and then we can't always know if there will be a TTL/Expiry.

[rphmeier]

requiring all future versions to have these fields really just makes topic and data (and potentially new fields) the only ones that can change. For backwards compatibility we can say that

• all versions of the message must begin with [expiry, ttl], and end with nonce.
• if we do need to change this for some reason, that's also fine. We just can't relay such messages to peers with older versions. The upgraded whisper nodes can choose to send old-style packets until they have enough peers who have upgraded as well.

It should still be against the protocol to send a peer a message they cannot evaluate expiry or PoW for. Otherwise, how can they relay it, or know when to stop relaying it?

[gluk256]

i see your point. so, do you want to remove Version?

[rphmeier]

clarify: may be empty. we need either that or a ping-pong mechanism

[gluk256]

ok

[rphmeier]

actually unnecessary: devp2p already handles protocol version negotiation. this only introduces further opportunity for mismatch

[rphmeier]

so then do we actually need a status packet at all?

[gluk256]

this is a legacy from all the old versions. actually, this is the only thing they have in common. this should be the first packet. it might possibly contain other information in future versions. i suggest we leave it for backwards compatibility. i will remove the version info, though.

[rphmeier]

floating point, 32 or 64 bit? You can serialize their bits directly, but you have to be careful when decoding that the bit pattern doesn't encode an sNAN. Maybe specify that sending any of qNAN, sNAN, INF, or -INF is illegal?

[rphmeier]

or perhaps even subnormal numbers shouldn't be allowed?

[gluk256]

I guess, it should be 32 bit, and meaningless stuff should not be allowed.

[rphmeier]

LGTM for now -- still pushing forward for an upgrade for multiple topics. Note that this can be done without breaking backwards compatibility or losing efficiency: if there is a single topic we RLP-encode as a single element, and otherwise as a list.

[rphmeier]

The next steps would involve an EIP for the "standard whisper payload format", which will specify stuff like flags, padding, supported encryption schemes, etc, and an EIP for the RPC API. These could be combined.

[rphmeier]

@gluk256 can you expand on the changes to the payload format and why they were necessary?

[pirapira]

Spelling: should be Whisper instead of Whipser.

[pirapira]

Grammar: should be be sent instead of be send.

[pirapira]

The first sentence says this packet contains two objects. The second sentence reads like the second object might not exist. Does this packet always contains two objects, or sometimes just one object?

[pirapira]

a is duplicated in a a 4-byte slice.

[gluk256]

thanks!

[yrashk]

What's the rationale behind reserving 128 codes?

[gluk256]

we need to decide upon some arbitrary number.
128 is as good as any other.

[yrashk]

@gluk256 this choice leads to integer overflow issues in certain circumstances (see openethereum/parity-ethereum#7567), so it'd be really great to figure out the motivation for such a wide space and if it is possible to reduce it.

[Arachnid]

This is a courtesy notice to let you know that the format for EIPs has been modified slightly. If you want your draft merged, you will need to make some small changes to how your EIP is formatted:

• Frontmatter is now contained between lines with only a triple dash ('---')
• Headers in the frontmatter are now lowercase.

If your PR is editing an existing EIP rather than creating a new one, this has already been done for you, and you need only rebase your PR.

In addition, a continuous build has been setup, which will check your PR against the rules for EIP formatting automatically once you update your PR. This build ensures all required headers are present, as well as performing a number of other checks.

Please rebase your PR against the latest master, and edit your PR to use the above format for frontmatter. For convenience, here's a sample header you can copy and adapt:

---
eip: <num>
title: <title>
author: <author>
type: [Standards Track|Informational|Meta]
category: [Core|Networking|Interface|ERC] (for type: Standards Track only)
status: Draft
created: <date>
---

[rphmeier]

just noticed that this was merged. the question of why the payload format changed from what was initially specified in this PR (commit
24c085a) was never answered. @gluk256 could you answer it now?