-
Notifications
You must be signed in to change notification settings - Fork 5.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add EIP: Forward compatible consensus data structures #8439
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
32b0a49
Add EIP: Forward compatible consensus data structures
etan-status 60ea790
Update EIPS/eip-7688.md
etan-status b6f9789
Update EIPS/eip-7688.md
etan-status 2325f7b
Update EIPS/eip-7688.md
etan-status 6e7ecbc
Apply suggestions from code review
etan-status e443fd9
clarifications regarding `Variant`
etan-status 3f7d2db
drop `Optional` type for historical features
etan-status 1be2457
add author
etan-status c78ccc9
Update EIPS/eip-7688.md
etan-status File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,296 @@ | ||
--- | ||
eip: 7688 | ||
title: Forward compatible consensus data structures | ||
description: Transition consensus SSZ data structures to StableContainer | ||
author: Etan Kissling (@etan-status), Cayman (@wemeetagain) | ||
discussions-to: https://ethereum-magicians.org/t/eip-7688-forward-compatible-consensus-data-structures/19673 | ||
status: Draft | ||
type: Standards Track | ||
category: Core | ||
created: 2024-04-15 | ||
requires: 7495 | ||
--- | ||
|
||
## Abstract | ||
|
||
This EIP defines the changes needed to adopt `StableContainer` from [EIP-7495](./eip-7495.md) in consensus data structures. | ||
|
||
## Motivation | ||
|
||
Ethereum's consensus data structures make heavy use of [Simple Serialize (SSZ)](https://github.com/ethereum/consensus-specs/blob/ef434e87165e9a4c82a99f54ffd4974ae113f732/ssz/simple-serialize.md) `Container`, which defines how they are serialized and merkleized. The merkleization scheme allows application implementations to verify that individual fields (and partial fields) have not been tampered with. This is useful, for example, in smart contracts of decentralized staking pools that wish to verify that participating validators have not been slashed. | ||
|
||
While SSZ `Container` defines how data structures are merkleized, the merkleization is prone to change across the different forks. When that happens, e.g., because new features are added or old features get removed, existing verifier implementations need to be updated to be able to continue processing proofs. | ||
|
||
`StableContainer`, of [EIP-7495](./eip-7495.md), is a forward compatible alternative that guarantees a forward compatible merkleization scheme. By transitioning consensus data structures to use `StableContainer`, smart contracts that contain verifier logic no longer have to be maintained in lockstep with Ethereum's fork schedule as long as the underlying features that they verify don't change. For example, as long as the concept of slashing is represented using the boolean `slashed` field, existing verifiers will not break when unrelated features get added or removed. This is also true for off-chain verifiers, e.g., in hardware wallets or in operating systems for mobile devices that are on a different software update cadence than Ethereum. | ||
|
||
## Specification | ||
|
||
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 and RFC 8174. | ||
|
||
### Conversion procedure | ||
|
||
For each converted data structure, a new `StableContainer` type `S` is introduced that serves as the primary definition of each data structure. | ||
|
||
- Each `StableContainer` is assigned a capacity to represent its potential design space that SHALL NOT change across future forks; if it is later determined that it is insufficient, a new field can be added to contain additional fields in a sub-container. | ||
- The `StableContainer` starts as a copy of the latest fork's `Container` equivalent. | ||
- To guarantee forward and backward compatibility, new fields from future forks MUST only be appended to the `StableContainer` definition. Existing fields MAY be converted to `Optional`. | ||
|
||
Furthermore, a `Variant` type is defined that is specific to the fork at which the conversion is applied. This `Variant` is the equivalent of the legacy `Container` type, except that it inherits from `Variant[S]`. The SSZ serialization of `Variant` is compatible with `Container`, but the merkleization and `hash_tree_root` are computed differently. Furthermore, `Variant` MAY use fields of `Optional` type if necessary. | ||
|
||
If subsequent forks add / remove fields, they specify a new `Variant`. This is similar to the previous approach of specifying a new `Container`. The `StableContainer` is the superset across all `Variant`; fields not present across all `Variant` use an `Optional` type. | ||
|
||
The fork specific `Variant` definitions use the same serialization as the legacy `Container` type. Only merkleization and `hash_tree_root` are computed differently when switching to the new scheme. | ||
|
||
### Limits | ||
|
||
| Name | Value | Description | | ||
| - | - | - | | ||
| `MAX_EXECUTION_PAYLOAD_FIELDS` | `uint64(2**6)` (= 64) | Maximum number of fields to which `ExecutionPayload` can ever grow in the future | | ||
| `MAX_BEACON_BLOCK_BODY_FIELDS` | `uint64(2**6)` (= 64) | Maximum number of fields to which `BeaconBlockBody` can ever grow in the future | | ||
| `MAX_BEACON_STATE_FIELDS` | `uint64(2**7)` (= 128) | Maximum number of fields to which `BeaconState` can ever grow in the future | | ||
|
||
### Fork-agnostic `StableContainer` definitions | ||
|
||
These type definitions are fork independent and shared across all forks. They are not exchanged over libp2p. | ||
|
||
```python | ||
class StableExecutionPayload(StableContainer[MAX_EXECUTION_PAYLOAD_FIELDS]): | ||
# Execution block header fields | ||
parent_hash: Hash32 | ||
fee_recipient: ExecutionAddress # 'beneficiary' in the yellow paper | ||
state_root: Bytes32 | ||
receipts_root: Bytes32 | ||
logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM] | ||
prev_randao: Bytes32 # 'difficulty' in the yellow paper | ||
block_number: uint64 # 'number' in the yellow paper | ||
gas_limit: uint64 | ||
gas_used: uint64 | ||
timestamp: uint64 | ||
extra_data: ByteList[MAX_EXTRA_DATA_BYTES] | ||
base_fee_per_gas: uint256 | ||
# Extra payload fields | ||
block_hash: Hash32 # Hash of execution block | ||
transactions: List[Transaction, MAX_TRANSACTIONS_PER_PAYLOAD] | ||
withdrawals: List[Withdrawal, MAX_WITHDRAWALS_PER_PAYLOAD] | ||
blob_gas_used: uint64 | ||
excess_blob_gas: uint64 | ||
... # Additional features from the fork that introduces this EIP | ||
|
||
class StableExecutionPayloadHeader(StableContainer[MAX_EXECUTION_PAYLOAD_FIELDS]): | ||
# Execution block header fields | ||
parent_hash: Hash32 | ||
fee_recipient: ExecutionAddress | ||
state_root: Bytes32 | ||
receipts_root: Bytes32 | ||
logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM] | ||
prev_randao: Bytes32 | ||
block_number: uint64 | ||
gas_limit: uint64 | ||
gas_used: uint64 | ||
timestamp: uint64 | ||
extra_data: ByteList[MAX_EXTRA_DATA_BYTES] | ||
base_fee_per_gas: uint256 | ||
# Extra payload fields | ||
block_hash: Hash32 # Hash of execution block | ||
transactions_root: Root | ||
withdrawals_root: Root | ||
blob_gas_used: uint64 | ||
excess_blob_gas: uint64 | ||
... # Additional features from the fork that introduces this EIP | ||
|
||
class StableBeaconBlockBody(StableContainer[MAX_BEACON_BLOCK_BODY_FIELDS]): | ||
randao_reveal: BLSSignature | ||
eth1_data: Eth1Data # Eth1 data vote | ||
graffiti: Bytes32 # Arbitrary data | ||
# Operations | ||
proposer_slashings: List[ProposerSlashing, MAX_PROPOSER_SLASHINGS] | ||
attester_slashings: List[AttesterSlashing, MAX_ATTESTER_SLASHINGS] | ||
attestations: List[Attestation, MAX_ATTESTATIONS] | ||
deposits: List[Deposit, MAX_DEPOSITS] | ||
voluntary_exits: List[SignedVoluntaryExit, MAX_VOLUNTARY_EXITS] | ||
sync_aggregate: SyncAggregate | ||
# Execution | ||
execution_payload: StableExecutionPayload | ||
bls_to_execution_changes: List[SignedBLSToExecutionChange, MAX_BLS_TO_EXECUTION_CHANGES] | ||
blob_kzg_commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK] | ||
... # Additional features from the fork that introduces this EIP | ||
|
||
class StableBeaconState(StableContainer[MAX_BEACON_STATE_FIELDS]): | ||
# Versioning | ||
genesis_time: uint64 | ||
genesis_validators_root: Root | ||
slot: Slot | ||
fork: Fork | ||
# History | ||
latest_block_header: BeaconBlockHeader | ||
block_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT] | ||
state_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT] | ||
historical_roots: List[Root, HISTORICAL_ROOTS_LIMIT] # Frozen in Capella, replaced by historical_summaries | ||
# Eth1 | ||
eth1_data: Eth1Data | ||
eth1_data_votes: List[Eth1Data, EPOCHS_PER_ETH1_VOTING_PERIOD * SLOTS_PER_EPOCH] | ||
eth1_deposit_index: uint64 | ||
# Registry | ||
validators: List[Validator, VALIDATOR_REGISTRY_LIMIT] | ||
balances: List[Gwei, VALIDATOR_REGISTRY_LIMIT] | ||
# Randomness | ||
randao_mixes: Vector[Bytes32, EPOCHS_PER_HISTORICAL_VECTOR] | ||
# Slashings | ||
slashings: Vector[Gwei, EPOCHS_PER_SLASHINGS_VECTOR] # Per-epoch sums of slashed effective balances | ||
# Participation | ||
previous_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT] | ||
current_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT] | ||
# Finality | ||
justification_bits: Bitvector[JUSTIFICATION_BITS_LENGTH] # Bit set for every recent justified epoch | ||
previous_justified_checkpoint: Checkpoint | ||
current_justified_checkpoint: Checkpoint | ||
finalized_checkpoint: Checkpoint | ||
# Inactivity | ||
inactivity_scores: List[uint64, VALIDATOR_REGISTRY_LIMIT] | ||
# Sync | ||
current_sync_committee: SyncCommittee | ||
next_sync_committee: SyncCommittee | ||
# Execution | ||
latest_execution_payload_header: StableExecutionPayloadHeader | ||
# Withdrawals | ||
next_withdrawal_index: WithdrawalIndex | ||
next_withdrawal_validator_index: ValidatorIndex | ||
# Deep history valid from Capella onwards | ||
historical_summaries: List[HistoricalSummary, HISTORICAL_ROOTS_LIMIT] | ||
... # Additional features from the fork that introduces this EIP | ||
``` | ||
|
||
### Fork-specific `Variant` definitions | ||
|
||
These type definitions are specific to the fork that introduces this EIP. They are used in place of the legacy fork-specific `Container` definitions. | ||
|
||
```python | ||
class ExecutionPayload(Variant[StableExecutionPayload]): | ||
# Execution block header fields | ||
parent_hash: Hash32 | ||
fee_recipient: ExecutionAddress # 'beneficiary' in the yellow paper | ||
state_root: Bytes32 | ||
receipts_root: Bytes32 | ||
logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM] | ||
prev_randao: Bytes32 # 'difficulty' in the yellow paper | ||
block_number: uint64 # 'number' in the yellow paper | ||
gas_limit: uint64 | ||
gas_used: uint64 | ||
timestamp: uint64 | ||
extra_data: ByteList[MAX_EXTRA_DATA_BYTES] | ||
base_fee_per_gas: uint256 | ||
# Extra payload fields | ||
block_hash: Hash32 # Hash of execution block | ||
transactions: List[Transaction, MAX_TRANSACTIONS_PER_PAYLOAD] | ||
withdrawals: List[Withdrawal, MAX_WITHDRAWALS_PER_PAYLOAD] | ||
blob_gas_used: uint64 | ||
excess_blob_gas: uint64 | ||
... # Additional features from the fork that introduces this EIP | ||
|
||
class ExecutionPayloadHeader(Variant[StableExecutionPayloadHeader]): | ||
# Execution block header fields | ||
parent_hash: Hash32 | ||
fee_recipient: ExecutionAddress | ||
state_root: Bytes32 | ||
receipts_root: Bytes32 | ||
logs_bloom: ByteVector[BYTES_PER_LOGS_BLOOM] | ||
prev_randao: Bytes32 | ||
block_number: uint64 | ||
gas_limit: uint64 | ||
gas_used: uint64 | ||
timestamp: uint64 | ||
extra_data: ByteList[MAX_EXTRA_DATA_BYTES] | ||
base_fee_per_gas: uint256 | ||
# Extra payload fields | ||
block_hash: Hash32 # Hash of execution block | ||
transactions_root: Root | ||
withdrawals_root: Root | ||
blob_gas_used: uint64 | ||
excess_blob_gas: uint64 | ||
etan-status marked this conversation as resolved.
Show resolved
Hide resolved
|
||
... # Additional features from the fork that introduces this EIP | ||
|
||
class BeaconBlockBody(Variant[StableBeaconBlockBody]): | ||
randao_reveal: BLSSignature | ||
eth1_data: Eth1Data # Eth1 data vote | ||
graffiti: Bytes32 # Arbitrary data | ||
# Operations | ||
proposer_slashings: List[ProposerSlashing, MAX_PROPOSER_SLASHINGS] | ||
attester_slashings: List[AttesterSlashing, MAX_ATTESTER_SLASHINGS] | ||
attestations: List[Attestation, MAX_ATTESTATIONS] | ||
deposits: List[Deposit, MAX_DEPOSITS] | ||
voluntary_exits: List[SignedVoluntaryExit, MAX_VOLUNTARY_EXITS] | ||
sync_aggregate: SyncAggregate | ||
# Execution | ||
execution_payload: ExecutionPayload | ||
bls_to_execution_changes: List[SignedBLSToExecutionChange, MAX_BLS_TO_EXECUTION_CHANGES] | ||
blob_kzg_commitments: List[KZGCommitment, MAX_BLOB_COMMITMENTS_PER_BLOCK] | ||
... # Additional features from the fork that introduces this EIP | ||
|
||
class BeaconState(Variant[StableBeaconState]): | ||
# Versioning | ||
genesis_time: uint64 | ||
genesis_validators_root: Root | ||
slot: Slot | ||
fork: Fork | ||
# History | ||
latest_block_header: BeaconBlockHeader | ||
block_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT] | ||
state_roots: Vector[Root, SLOTS_PER_HISTORICAL_ROOT] | ||
historical_roots: List[Root, HISTORICAL_ROOTS_LIMIT] # Frozen in Capella, replaced by historical_summaries | ||
# Eth1 | ||
eth1_data: Eth1Data | ||
eth1_data_votes: List[Eth1Data, EPOCHS_PER_ETH1_VOTING_PERIOD * SLOTS_PER_EPOCH] | ||
eth1_deposit_index: uint64 | ||
# Registry | ||
validators: List[Validator, VALIDATOR_REGISTRY_LIMIT] | ||
balances: List[Gwei, VALIDATOR_REGISTRY_LIMIT] | ||
# Randomness | ||
randao_mixes: Vector[Bytes32, EPOCHS_PER_HISTORICAL_VECTOR] | ||
# Slashings | ||
slashings: Vector[Gwei, EPOCHS_PER_SLASHINGS_VECTOR] # Per-epoch sums of slashed effective balances | ||
# Participation | ||
previous_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT] | ||
current_epoch_participation: List[ParticipationFlags, VALIDATOR_REGISTRY_LIMIT] | ||
# Finality | ||
justification_bits: Bitvector[JUSTIFICATION_BITS_LENGTH] # Bit set for every recent justified epoch | ||
previous_justified_checkpoint: Checkpoint | ||
current_justified_checkpoint: Checkpoint | ||
finalized_checkpoint: Checkpoint | ||
# Inactivity | ||
inactivity_scores: List[uint64, VALIDATOR_REGISTRY_LIMIT] | ||
# Sync | ||
current_sync_committee: SyncCommittee | ||
next_sync_committee: SyncCommittee | ||
# Execution | ||
latest_execution_payload_header: ExecutionPayloadHeader | ||
# Withdrawals | ||
next_withdrawal_index: WithdrawalIndex | ||
next_withdrawal_validator_index: ValidatorIndex | ||
# Deep history valid from Capella onwards | ||
historical_summaries: List[HistoricalSummary, HISTORICAL_ROOTS_LIMIT] | ||
... # Additional features from the fork that introduces this EIP | ||
``` | ||
|
||
## Rationale | ||
|
||
### Best timing? | ||
|
||
Applying this EIP breaks `hash_tree_root` and Merkle tree verifiers a single time, while promising forward compatibility from the fork going forward. It is best to apply it before merkleization would be broken by different changes. Merkleization is broken by a `Container` reaching a new power of 2 in its number of fields. | ||
|
||
### Can this be applied retroactively? | ||
|
||
While `Variant` serializes in the same way as the legacy `Container`, the merkleization and `hash_tree_root` of affected data structures changes. Therefore, verifiers that wish to process Merkle proofs of legacy variants still need to support the corresponding legacy schemes. | ||
|
||
## Backwards Compatibility | ||
|
||
Existing Merkle proof verifiers need to be updated to support the new Merkle tree shape. This includes verifiers in smart contracts on different blockchains and verifiers in hardware wallets, if applicable. | ||
|
||
Note that backwards compatibility is also broken when one of the converted `Container` data structures would reach a new power of 2 in its number of fields. | ||
|
||
## Security Considerations | ||
|
||
None | ||
|
||
## Copyright | ||
|
||
Copyright and related rights waived via [CC0](../LICENSE.md). |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You might mention that a new Variant will be defined for every additional fork as needed, as fields are added or removed.