HIP-1056: Block Streams

HIP/hip-1056.md

@@ -59,7 +59,7 @@ additional block and state APIs that benefit the Hedera community (e.g., state p
 "downstream service" that can consume block streams.
 
 TSS signatures refer to a single, aggregated TSS-BLS signature that verifies a block has been signed by a threshold of
-Hedera consensus weight. Each block will include TSS signatures, offering verifiable assurance that the block was
+Hedera consensus stake weight. Each block will include TSS signatures, offering verifiable assurance that the block was
 produced by a configured threshold of the network stake weight.
 
 Subsequent HIPs, to be introduced at a later date, will provide details on Block Nodes and TSS Signatures.
@@ -188,7 +188,7 @@ transaction execution. These are hashed separately into two different *block str
 we can separate out a subset of historical blocks with just their input items, or just output items. That subset block stream could then be used
 for transaction replay or transaction execution verification, for example.
 
-![Block Stream Merkle Tree](../assets/hip-blocks/block-stream-merkle.png)
+![Block Stream Merkle Tree](../assets/hip-blocks/block-stream-merkle.svg)
 
 Consensus nodes have a `state merkle tree`, which is a merkle tree containing the current state of the consensus node.
 The block stream has a separate `block stream merkle tree` which is a merkle tree of all the items in the block stream
@@ -246,7 +246,7 @@ full complexity of the merkle state tree.
    can authenticate transactional data without the cost and complexity of downloading multiple signature files for
    verification.
 3. As a verifier for a network, I want a single, self-contained stream of data with aggregated signature signed by a
-   threshold weight of nodes, so that I can cost-effectively confirm that the stream represents the consensus output of
+   threshold of stake weights, so that I can cost-effectively confirm that the stream represents the consensus output of
    the network.
 4. As a downstream developer, I want a low latency feed that provides the state changes associated with transaction(s)
    within a block, so that I can interact with state changes with the smallest possible delay.
@@ -1068,7 +1068,7 @@ message SingletonUpdateChange {
          * A change to the platform state singleton.
          */
         com.hedera.hapi.platform.state.PlatformState platform_state_value = 12;
-        
+
         /**
          * A change to the roster state singleton.
          */
@@ -1579,7 +1579,7 @@ message MerkleSiblingHash {
 
 The following diagram illustrates what the 4 sub trees contain and how blocks are chained.
 
-![Block Stream Merkle](../assets/hip-blocks/block-stream-merkle.png)
+![Block Stream Merkle](../assets/hip-blocks/block-stream-merkle.svg)
 
 The input and output item merkle trees are balanced base 2 size binary trees. The leaves across the bottom are the items in the order they occur in the block. If the total number of input or output items contained in the block is less than a power of 2 then the empty leaves are assumed to be `null` (zero length bytes). The nature of the merkle tree creation logic allows for the consensus node or block validator to calculate the input and output tree merkle hashes as items are received in a streaming way.
 
@@ -1730,11 +1730,38 @@ clients consuming a block stream.
 ## Rejected Ideas
 
 ### Re-writing pre block stream transaction body protobufs
-The previous record streams protobuf contained deprecated fields and messages that could be better grouped for efficiency and consistency.
-With a new data stream there was a consideration to rewrite the previous format.
-Unfortunately, the TransactionBody message itself cannot be re-written due to signing verification implications on backwards compatability.
-This is because mirror node and other products need to be able to parse historical transaction bytes utilizing older formats and verify them.
-However, the transaction wrappers in the old formats could still be optimized. This may be explored in the future.
+The previous record streams protobuf contained deprecated fields and messages that could be better grouped for efficiency
+and consistency. With a new data stream there was a consideration to rewrite the previous format.
+Unfortunately, the TransactionBody message itself cannot be re-written due to signing verification implications on backwards
+compatability. This is because mirror node and other products need to be able to parse historical transaction bytes utilizing
+older formats and verify them. However, the transaction wrappers in the old formats could still be optimized. This may be
+explored in the future.
+
+### Ensuring all events across upgrade boundaries are captured
+The block streams design ensures that all events are captured and that blocks are signed. However, how does one denote the last
+block before upgrade if the act of signing a block `n` produces new events that must be recorded in block `n+1`?
+One consideration was to create a marker after which a future block may be ignored in terms of signing requirements. Though this
+could work it would result in Blocks that are unsigned even if they are ignored. This also makes stream consideration by clients
+a bit more complex as they would have to be aware of and handle this marker case.
+The final solution was to capture events created by signing Block `n` in the Pre Consensus Event Stream (PCES) across the upgrade
+boundary for insertion into a future Block `n+1` after upgrade.
+
+#### Impact of large entities when externalizing individual entity state. 
+The Block Stream plans to ensure visibilty of all modified entities by externalizing the full entity state in the stream.
+This is to solve the issue of partial data in record streams where a client may miss the externalization of state data and would
+therefore possess a partial state for entities with few easy options to retrieve the missing data.
+By externalizing the full entity state in tehe block stream a client can get the complete entity details on any update and won't have
+to employ upsert logic. An emergent feature when externalizing the full entity state of modified entities is that entities who's
+properties are modified multiple times would be externalized in the stream multiple times.
+These accounts may have multiple keys and other potentially large list properties (e.g. approvals) that may result in a large block size.
+To manage this an idea inspired from video iframe approach (when the image changes the whole entity is sent, if not only diffs
+are sent) was considered. In this case every entity could have a last updated counter that is updated on entity changes, if the
+value is over a certain threshold then the whole entity is externalized, if not only deltas would be externalized. This would optimize
+how far back a client would have to go to retrieve state info. However, this approach would require the counter to be stored in state
+for every entity which has undesired impacts on state size.
+Another inspired approach is that on the first observation of an entity in a block would cause the full entity to be extenralized,
+after that only the delta changes are externalized for the remainder of the block.
+In this way any consumer of a block would have the complete state by applying the diffs onto the initial state.
 
 ## Copyright/license