Skip to content

Commit f77ca3b

Browse files
ccecce
committed
improved README
1 parent b0ff3f2 commit f77ca3b

File tree

1 file changed

+33
-32
lines changed

1 file changed

+33
-32
lines changed

‎network/vpack/README.md‎

Lines changed: 33 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,9 @@ Vote compression uses two layers:
1212

1313
1. **Stateless compression** (`StatelessEncoder`/`StatelessDecoder`): Removes msgpack formatting and field names, replacing them with a bitmask. This is always applied and has no memory overhead.
1414

15-
2. **Stateful compression (optional)** (`StatefulEncoder`/`StatefulDecoder`): Further compresses by replacing frequently repeated values with references to LRU tables and a sliding window. This layer requires per-connection state (configurable size, e.g., ~224KB per encoder/decoder pair for 1024-entry tables) and is optional. When used, it operates on the output of the stateless layer.
15+
2. **Stateful compression (optional)** (`StatefulEncoder`/`StatefulDecoder`): Further compresses by replacing frequently repeated values with references to LRU tables and a sliding window. This layer requires per-connection state (configurable size, e.g., ~224KB per direction for 1024-entry tables) and is optional. When used, it operates on the output of the stateless layer.
1616

17-
Both layers use the same 2-byte header format, with byte 0 used by the stateless layer and byte 1 used by the stateful layer (zero when stateful compression is not used).
17+
Both layers use the same 2-byte header format, with byte 0 used by the stateless layer, and byte 1 used by the stateful layer.
1818

1919
---
2020

@@ -60,42 +60,43 @@ Integers use msgpack's variable-length unsigned integer encoding:
6060

6161
### 2.2 Stateful compression flags (byte 1)
6262

63-
When stateful compression is used, byte 1 encodes which values have been replaced by references:
63+
When stateful compression is used, byte 1 encodes which values have been replaced by references. When stateful compression is not used, byte 1 must be zero.
6464

65-
| Bits | Field | Meaning |
66-
| ---- | ---------------- | ----------------------------------------------------------------- |
67-
| 0-1 | `rnd` encoding | `00`=literal, `01`=+1, `10`=-1, `11`=same as last round |
68-
| 2-4 | `prop` reference | `000`=literal, `001`-`111`=sliding window index (1-7) |
69-
| 5 | `snd` reference | `0`=literal (32 bytes), `1`=LRU table reference (2 bytes) |
70-
| 6 | `pk` reference | `0`=literal (96 bytes), `1`=LRU table reference (2 bytes) |
71-
| 7 | `pk2` reference | `0`=literal (96 bytes), `1`=LRU table reference (2 bytes) |
65+
| Bits | Flag | Field(s) | Meaning |
66+
| ---- | ---------- | ---------------------------------- | ----------------------------------------------------------- |
67+
| 0-1 | `rndDelta` | `r.rnd` delta encoding | `00`=literal, `01`=+1, `10`=-1, `11`=same as last round |
68+
| 2-4 | `propRef` | `r.prop` window reference | `000`=literal, `001`-`111`=sliding window index (1-7) |
69+
| 5 | `sndRef` | `r.snd` reference | `0`=literal (32 bytes), `1`=LRU table reference (2 bytes) |
70+
| 6 | `pkRef` | `sig.p` + `sig.p1s` reference | `0`=literal (96 bytes), `1`=LRU table reference (2 bytes) |
71+
| 7 | `pk2Ref` | `sig.p2` + `sig.p2s` reference | `0`=literal (96 bytes), `1`=LRU table reference (2 bytes) |
7272

7373
The stateful layer uses:
74-
- **Round delta encoding**: Most votes reference the same round or adjacent rounds, so 2 bits can encode common cases.
74+
- **Round delta encoding**: Most votes reference the same round as before, with some interleaving votes between the current and next round, as voters gradually observe consensus and move on to the next round. A 2-bit encoding specifies whether `r.rnd` is the same as the previous vote, or has increased or decreased by one, and can be omitted from the message for these cases. If `rndDelta` is `00`, the literal value appears in the message.
7575
- **Proposal sliding window**: A 7-entry HPACK-style window tracks recent proposal values. In a typical round, all votes should be for the same proposal value, compressing previously-seen proposal values from ~96 bytes down to a 3-bit reference.
76-
- **LRU tables**: Three 2-way set-associative hash tables cache recently seen values for `snd` (sender addresses), `pk` (public key + signature pairs), and `pk2` (second-tier key + signature pairs). Since some consensus participants (and their associated public keys) will appear more often than others based on the distribution of stake, a larger table can capture participant-related data and replace them with LRU references. For each participant, the `snd` field is re-used across all votes; the `pk` reference is re-used across all votes in the same round; and the `pk2` reference is re-used across all votes in a batch (typically thousands of rounds) under our hierarchical 3-tiered consensus signature scheme.
77-
78-
When stateful compression is not used, byte 1 must be zero.
76+
- **LRU tables**: Three 2-way set-associative hash tables cache recently seen values for sender addresses, and two tiers of (public key, signature) pairs. Since some consensus participants vote in rounds more often than others, these tables record the most common field values re-used across votes by the same participant, and replace them with references:
77+
- The `snd` table tracks participating addresses, which are re-used across all votes by a participant;
78+
- the `pk` table tracks `sig.pk` and `sig.p1s` values, which are re-used across votes in the same round by a participant;
79+
- and the `pk2` table tracks `sig.p2` and `sig.p2s` values, which are re-used across all votes in a batch by a participant (typically thousands of rounds), under Algorand's hierarchical consensus signature scheme.
7980

8081
---
8182

8283
## 3. Field serialization order
8384

84-
After the 2-byte header, the encoder emits values in the following order:
85-
86-
| Field | Type | Encoded size | Presence flag |
87-
| -------------- | ------------------------------ | ------------ | ------------- |
88-
| `pf` | VRF credential | 80 bytes | Required |
89-
| `r.per` | Period | varuint | `bitPer` |
90-
| `r.prop.dig` | Proposal digest | 32 bytes | `bitDig` |
91-
| `r.prop.encdig`| Digest of encoded proposal | 32 bytes | `bitEncDig` |
92-
| `r.prop.oper` | Proposal's original period | varuint | `bitOper` |
93-
| `r.prop.oprop` | Proposal's original proposer | 32 bytes | `bitOprop` |
94-
| `r.rnd` | Round number | varuint | Required |
95-
| `r.snd` | Voter's (sender) address | 32 bytes | Required |
96-
| `r.step` | Step | varuint | `bitStep` |
97-
| `sig.p` | Ed25519 public key | 32 bytes | Required |
98-
| `sig.p1s` | Signature over offset ID | 64 bytes | Required |
99-
| `sig.p2` | Second-tier Ed25519 public key | 32 bytes | Required |
100-
| `sig.p2s` | Signature over batch ID | 64 bytes | Required |
101-
| `sig.s` | Signature over vote using `p` | 64 bytes | Required |
85+
After the 2-byte header, the encoder emits values in the following order. If stateful compression is enabled, the "stateful encoding" column specifies how each field is additionally transformed or omitted.
86+
87+
| Field | Type | Encoded size | Presence flag | Stateful encoding |
88+
| -------------- | ------------------------------ | ------------ | ------------- | -------------------------------- |
89+
| `pf` | VRF credential | 80 bytes | Required | Unchanged |
90+
| `r.per` | Period | varuint | `bitPer` | Unchanged |
91+
| `r.prop.dig` | Proposal digest | 32 bytes | `bitDig` | Omitted if `propRef` set |
92+
| `r.prop.encdig`| Digest of encoded proposal | 32 bytes | `bitEncDig` | Omitted if `propRef` set |
93+
| `r.prop.oper` | Proposal's original period | varuint | `bitOper` | Omitted if `propRef` set |
94+
| `r.prop.oprop` | Proposal's original proposer | 32 bytes | `bitOprop` | Omitted if `propRef` set |
95+
| `r.rnd` | Round number | varuint | Required | Omitted if `rndDelta` set |
96+
| `r.snd` | Voter's (sender) address | 32 bytes | Required | 2-byte reference if `sndRef` set |
97+
| `r.step` | Step | varuint | `bitStep` | literal (if present) |
98+
| `sig.p` | Ed25519 public key | 32 bytes | Required | 2-byte reference if `pkRef` set |
99+
| `sig.p1s` | Signature over offset ID | 64 bytes | Required | Omitted if `pkRef` set |
100+
| `sig.p2` | Second-tier Ed25519 public key | 32 bytes | Required | 2-byte reference if `pk2Ref` set |
101+
| `sig.p2s` | Signature over batch ID | 64 bytes | Required | Omitted if `pk2Ref` set |
102+
| `sig.s` | Signature over vote using `p` | 64 bytes | Required | Unchanged |

0 commit comments

Comments
 (0)