Merge branch 'dev' into pr2965

README.md

@@ -19,7 +19,7 @@ Features are researched and developed in parallel, and then consolidated into se
 | - | - | - | - |
 | 0 | **Phase0** |`0` | <ul><li>Core</li><ul><li>[The beacon chain](specs/phase0/beacon-chain.md)</li><li>[Deposit contract](specs/phase0/deposit-contract.md)</li><li>[Beacon chain fork choice](specs/phase0/fork-choice.md)</li></ul><li>Additions</li><ul><li>[Honest validator guide](specs/phase0/validator.md)</li><li>[P2P networking](specs/phase0/p2p-interface.md)</li><li>[Weak subjectivity](specs/phase0/weak-subjectivity.md)</li></ul></ul> |
 | 1 |  **Altair** | `74240` | <ul><li>Core</li><ul><li>[Beacon chain changes](specs/altair/beacon-chain.md)</li><li>[Altair fork](specs/altair/fork.md)</li></ul><li>Additions</li><ul><li>[Light client sync protocol](specs/altair/light-client/sync-protocol.md) ([full node](specs/altair/light-client/full-node.md), [light client](specs/altair/light-client/light-client.md), [networking](specs/altair/light-client/p2p-interface.md))</li><li>[Honest validator guide changes](specs/altair/validator.md)</li><li>[P2P networking](specs/altair/p2p-interface.md)</li></ul></ul> |
-| 2 | **Bellatrix** <br/> (["The Merge"](https://ethereum.org/en/upgrades/merge/)) | TBD | <ul><li>Core</li><ul><li>[Beacon Chain changes](specs/bellatrix/beacon-chain.md)</li><li>[Bellatrix fork](specs/bellatrix/fork.md)</li><li>[Fork choice changes](specs/bellatrix/fork-choice.md)</li></ul><li>Additions</li><ul><li>[Honest validator guide changes](specs/bellatrix/validator.md)</li><li>[P2P networking](specs/bellatrix/p2p-interface.md)</li></ul></ul> |
+| 2 | **Bellatrix** <br/> (["The Merge"](https://ethereum.org/en/upgrades/merge/)) | `144896` | <ul><li>Core</li><ul><li>[Beacon Chain changes](specs/bellatrix/beacon-chain.md)</li><li>[Bellatrix fork](specs/bellatrix/fork.md)</li><li>[Fork choice changes](specs/bellatrix/fork-choice.md)</li></ul><li>Additions</li><ul><li>[Honest validator guide changes](specs/bellatrix/validator.md)</li><li>[P2P networking](specs/bellatrix/p2p-interface.md)</li></ul></ul> |
 
 ### In-development Specifications
 | Code Name or Topic | Specs | Notes |

configs/mainnet.yaml

@@ -12,8 +12,8 @@ CONFIG_NAME: 'mainnet'
 
 # Transition
 # ---------------------------------------------------------------
-# TBD, 2**256-2**10 is a placeholder
-TERMINAL_TOTAL_DIFFICULTY: 115792089237316195423570985008687907853269984665640564039457584007913129638912
+# Estimated on Sept 15, 2022
+TERMINAL_TOTAL_DIFFICULTY: 58750000000000000000000
 # By default, don't use these params
 TERMINAL_BLOCK_HASH: 0x0000000000000000000000000000000000000000000000000000000000000000
 TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH: 18446744073709551615
@@ -43,7 +43,7 @@ ALTAIR_FORK_VERSION: 0x01000000
 ALTAIR_FORK_EPOCH: 74240  # Oct 27, 2021, 10:56:23am UTC
 # Bellatrix
 BELLATRIX_FORK_VERSION: 0x02000000
-BELLATRIX_FORK_EPOCH: 18446744073709551615
+BELLATRIX_FORK_EPOCH: 144896  # Sept 6, 2022, 11:34:47am UTC
 # Capella
 CAPELLA_FORK_VERSION: 0x03000000
 CAPELLA_FORK_EPOCH: 18446744073709551615

configs/minimal.yaml

@@ -12,7 +12,7 @@ CONFIG_NAME: 'minimal'
 
 # Transition
 # ---------------------------------------------------------------
-# TBD, 2**256-2**10 is a placeholder
+# 2**256-2**10 for testing minimal network
 TERMINAL_TOTAL_DIFFICULTY: 115792089237316195423570985008687907853269984665640564039457584007913129638912
 # By default, don't use these params
 TERMINAL_BLOCK_HASH: 0x0000000000000000000000000000000000000000000000000000000000000000

presets/minimal/phase0.yaml

@@ -4,11 +4,11 @@
 # ---------------------------------------------------------------
 # [customized] Just 4 committees for slot for testing purposes
 MAX_COMMITTEES_PER_SLOT: 4
-# [customized] unsecure, but fast
+# [customized] insecure, but fast
 TARGET_COMMITTEE_SIZE: 4
 # 2**11 (= 2,048)
 MAX_VALIDATORS_PER_COMMITTEE: 2048
-# [customized] Faster, but unsecure.
+# [customized] Faster, but insecure.
 SHUFFLE_ROUND_COUNT: 10
 # 4
 HYSTERESIS_QUOTIENT: 4

setup.py

@@ -586,6 +586,7 @@ def imports(cls, preset_name: str):
         return super().imports(preset_name) + f'''
 from eth2spec.utils import kzg
 from eth2spec.bellatrix import {preset_name} as bellatrix
+from eth2spec.utils.ssz.ssz_impl import serialize as ssz_serialize
 '''
 
     @classmethod

specs/bellatrix/beacon-chain.md

@@ -91,7 +91,7 @@ Bellatrix updates a few configuration values to move penalty parameters to their
 
 | Name | Value |
 | - | - |
-| `TERMINAL_TOTAL_DIFFICULTY` | **TBD** |
+| `TERMINAL_TOTAL_DIFFICULTY` | `58750000000000000000000` (Estimated: Sept 15, 2022)|
 | `TERMINAL_BLOCK_HASH` | `Hash32()` |
 | `TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH` | `FAR_FUTURE_EPOCH` |
 
@@ -298,8 +298,6 @@ def slash_validator(state: BeaconState,
     increase_balance(state, whistleblower_index, Gwei(whistleblower_reward - proposer_reward))
 ```
 
-
-
 ## Beacon chain state transition function
 
 ### Execution engine

specs/bellatrix/fork.md

@@ -29,7 +29,7 @@ Warning: this configuration is not definitive.
 | Name | Value |
 | - | - |
 | `BELLATRIX_FORK_VERSION` | `Version('0x02000000')` |
-| `BELLATRIX_FORK_EPOCH` | `Epoch(18446744073709551615)` **TBD** |
+| `BELLATRIX_FORK_EPOCH` | `Epoch(144896)` (Sept 6, 2022, 11:34:47am UTC) |
 
 ## Helper functions
 

specs/eip4844/p2p-interface.md

@@ -104,7 +104,7 @@ The following validations MUST pass before forwarding the `signed_blobs_sidecar`
 Alias `sidecar = signed_blobs_sidecar.message`.
 - _[IGNORE]_ the `sidecar.beacon_block_slot` is for the current slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) -- i.e. `sidecar.beacon_block_slot == current_slot`.
 - _[REJECT]_ the `sidecar.blobs` are all well formatted, i.e. the `BLSFieldElement` in valid range (`x < BLS_MODULUS`).
-- _[REJECT]_ The KZG proof is a correctly encoded compressed BLS G1 Point -- i.e. `bls.KeyValidate(blobs_sidecar.kzg_aggregated_proof)
+- _[REJECT]_ The KZG proof is a correctly encoded compressed BLS G1 Point -- i.e. `bls.KeyValidate(blobs_sidecar.kzg_aggregated_proof)`
 - _[REJECT]_ the beacon proposer signature, `signed_blobs_sidecar.signature`, is valid -- i.e.
     - Let `domain = get_domain(state, DOMAIN_BLOBS_SIDECAR, sidecar.beacon_block_slot // SLOTS_PER_EPOCH)`
     - Let `signing_root = compute_signing_root(sidecar, domain)`

specs/eip4844/validator.md

@@ -93,9 +93,10 @@ def is_data_available(slot: Slot, beacon_block_root: Root, blob_kzg_commitments:
 ```python
 def hash_to_bls_field(x: Container) -> BLSFieldElement:
     """
-    This function is used to generate Fiat-Shamir challenges. The output is not uniform over the BLS field.
+    Compute 32-byte hash of serialized container and convert it to BLS field.
+    The output is not uniform over the BLS field.
     """
-    return int.from_bytes(hash_tree_root(x), "little") % BLS_MODULUS
+    return int.from_bytes(hash(ssz_serialize(x)), "little") % BLS_MODULUS
 ```
 
 ### `compute_powers`

specs/phase0/p2p-interface.md

@@ -1327,7 +1327,7 @@ Requests are segregated by protocol ID to:
    1. This has the benefit that clients can explicitly choose which RFCs to deploy
       without buying into all other RFCs that may be included in that top-level version.
    2. Affording this level of granularity with a top-level protocol would imply creating as many variants
-      (e.g. /protocol/43-{a,b,c,d,...}) as the cartesian product of RFCs inflight, O(n^2).
+      (e.g. /protocol/43-{a,b,c,d,...}) as the cartesian product of RFCs in-flight, O(n^2).
 7. Allow us to simplify the payload of requests.
   Request-id’s and method-ids no longer need to be sent.
   The encoding/request type and version can all be handled by the framework.

sync/optimistic.md

@@ -1,5 +1,38 @@
 # Optimistic Sync
 
+## Table of contents
+<!-- TOC -->
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+
+- [Introduction](#introduction)
+- [Constants](#constants)
+- [Helpers](#helpers)
+- [Mechanisms](#mechanisms)
+  - [When to optimistically import blocks](#when-to-optimistically-import-blocks)
+  - [How to optimistically import blocks](#how-to-optimistically-import-blocks)
+  - [How to apply `latestValidHash` when payload status is `INVALID`](#how-to-apply-latestvalidhash-when-payload-status-is-invalid)
+  - [Execution Engine Errors](#execution-engine-errors)
+  - [Assumptions about Execution Engine Behaviour](#assumptions-about-execution-engine-behaviour)
+  - [Re-Orgs](#re-orgs)
+- [Fork Choice](#fork-choice)
+  - [Fork Choice Poisoning](#fork-choice-poisoning)
+- [Checkpoint Sync (Weak Subjectivity Sync)](#checkpoint-sync-weak-subjectivity-sync)
+- [Validator assignments](#validator-assignments)
+  - [Block Production](#block-production)
+  - [Attesting](#attesting)
+  - [Participating in Sync Committees](#participating-in-sync-committees)
+- [Ethereum Beacon APIs](#ethereum-beacon-apis)
+- [Design Decision Rationale](#design-decision-rationale)
+  - [Why sync optimistically?](#why-sync-optimistically)
+  - [Why `SAFE_SLOTS_TO_IMPORT_OPTIMISTICALLY`?](#why-safe_slots_to_import_optimistically)
+  - [Transitioning from VALID -> INVALIDATED or INVALIDATED -> VALID](#transitioning-from-valid---invalidated-or-invalidated---valid)
+  - [What about Light Clients?](#what-about-light-clients)
+  - [What if `TERMINAL_BLOCK_HASH` is used?](#what-if-terminal_block_hash-is-used)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+<!-- /TOC -->
+
 ## Introduction
 
 In order to provide a syncing execution engine with a partial view of the head
@@ -89,8 +122,12 @@ def is_optimistic_candidate_block(opt_store: OptimisticStore, current_slot: Slot
     return False
 ```
 
-Let only a node which returns `is_optimistic(opt_store, head) is True` be an *optimistic
-node*. Let only a validator on an optimistic node be an *optimistic validator*.
+Let a node be an *optimistic node* if its fork choice is in one of the following states:
+1. `is_optimistic(opt_store, head) is True`
+2. Blocks from every viable (with respect to FFG) branch have transitioned from `NOT_VALIDATED` to `INVALIDATED`
+leaving the block tree without viable branches
+
+Let only a validator on an optimistic node be an *optimistic validator*.
 
 When this specification only defines behaviour for an optimistic
 node/validator, but *not* for the non-optimistic case, assume default
@@ -163,6 +200,23 @@ the merge block MUST be treated the same as
 an `INVALIDATED` block (i.e., it and all its descendants are invalidated and
 removed from the block tree).
 
+### How to apply `latestValidHash` when payload status is `INVALID`
+
+Processing an `INVALID` payload status depends on the `latestValidHash` parameter.
+The general approach is as follows:
+1. Consensus engine MUST identify `invalidBlock` as per definition in the table below.
+2. `invalidBlock` and all of its descendants MUST be transitioned from `NOT_VALIDATED` to `INVALIDATED`.
+
+| `latestValidHash` | `invalidBlock` |
+|:- |:- |
+| Execution block hash | The *child* of a block with `body.execution_payload.block_hash == latestValidHash` in the chain containing the block with payload in question |
+| `0x00..00` (all zeroes) | The first block with `body.execution_payload != ExecutionPayload()` in the chain containing a block with payload in question |
+| `null` | Block with payload in question |
+
+When `latestValidHash` is a meaningful execution block hash but consensus engine
+cannot find a block satisfying `body.execution_payload.block_hash == latestValidHash`,
+consensus engine SHOULD behave the same as if `latestValidHash` was `null`.
+
 ### Execution Engine Errors
 
 When an execution engine returns an error or fails to respond to a payload
@@ -270,6 +324,28 @@ optimistic blocks (and vice-versa).
 
 ## Design Decision Rationale
 
+### Why sync optimistically?
+
+Most execution engines use state sync as a default sync mechanism on Ethereum Mainnet 
+because executing blocks from genesis takes several weeks on commodity hardware.
+
+State sync requires the knowledge of the current head of the chain to converge eventually.
+If not constantly fed with the most recent head, state sync won't be able to complete
+because the recent state soon becomes unavailable due to state trie pruning.
+
+Optimistic block import (i.e. import when the execution engine *cannot* currently validate the payload)
+breaks a deadlock between the execution layer sync process and importing beacon blocks
+while the execution engine is syncing.
+
+Optimistic sync is also an optimal strategy for execution engines using block execution as a default
+sync mechanism (e.g. Erigon). Alternatively, a consensus engine may inform the execution engine with a payload
+obtained from a checkpoint block, then wait until the execution layer catches up with it and proceed
+in lock step after that. This alternative approach would keep user in limbo for several hours and
+would increase time of the sync process as batch sync has more opportunities for optimisation than the lock step.
+
+Aforementioned premises make optimistic sync a *generalized* solution for interaction between consensus and
+execution engines during the sync process.
+
 ### Why `SAFE_SLOTS_TO_IMPORT_OPTIMISTICALLY`?
 
 Nodes can only import an optimistic block if their justified checkpoint is
@@ -328,7 +404,7 @@ specification since it's only possible with a faulty EE.
 
 Such a scenario requires manual intervention.
 
-## What about Light Clients?
+### What about Light Clients?
 
 An alternative to optimistic sync is to run a light client inside/alongside
 beacon nodes that mitigates the need for optimistic sync by providing
@@ -340,7 +416,7 @@ A notable thing about optimistic sync is that it's *optional*. Should an
 implementation decide to go the light-client route, then they can just ignore
 optimistic sync all together.
 
-## What if `TERMINAL_BLOCK_HASH` is used?
+### What if `TERMINAL_BLOCK_HASH` is used?
 
 If the terminal block hash override is used (i.e., `TERMINAL_BLOCK_HASH !=
 Hash32()`), the [`validate_merge_block`](../specs/bellatrix/fork-choice.md#validate_merge_block)

tests/core/pyspec/eth2spec/VERSION.txt

@@ -1 +1 @@
-1.2.0-rc.1
+1.2.0-rc.3

tests/core/pyspec/eth2spec/test/altair/transition/test_transition.py

@@ -3,6 +3,9 @@
     ForkMeta,
     with_fork_metas,
 )
+from eth2spec.test.helpers.random import (
+    randomize_state,
+)
 from eth2spec.test.helpers.constants import (
     ALL_PRE_POST_FORKS,
 )
@@ -17,9 +20,31 @@
     skip_slots,
     state_transition_across_slots,
     transition_to_next_epoch_and_append_blocks,
+    transition_until_fork,
 )
 
 
+@with_fork_metas([ForkMeta(pre_fork_name=pre, post_fork_name=post, fork_epoch=2) for pre, post in ALL_PRE_POST_FORKS])
+def test_simple_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag):
+    transition_until_fork(spec, state, fork_epoch)
+
+    # check pre state
+    assert spec.get_current_epoch(state) < fork_epoch
+
+    yield "pre", state
+
+    # irregular state transition to handle fork:
+    blocks = []
+    state, block = do_fork(state, spec, post_spec, fork_epoch)
+    blocks.append(post_tag(block))
+
+    # continue regular state transition with new spec into next epoch
+    transition_to_next_epoch_and_append_blocks(post_spec, state, post_tag, blocks, only_last_block=True)
+
+    yield "blocks", blocks
+    yield "post", state
+
+
 @with_fork_metas([ForkMeta(pre_fork_name=pre, post_fork_name=post, fork_epoch=2) for pre, post in ALL_PRE_POST_FORKS])
 def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag):
     """
@@ -56,6 +81,37 @@ def test_normal_transition(state, fork_epoch, spec, post_spec, pre_tag, post_tag
     yield "post", state
 
 
+@with_fork_metas([ForkMeta(pre_fork_name=pre, post_fork_name=post, fork_epoch=8) for pre, post in ALL_PRE_POST_FORKS])
+def test_transition_randomized_state(state, fork_epoch, spec, post_spec, pre_tag, post_tag):
+    randomize_state(spec, state)
+
+    transition_until_fork(spec, state, fork_epoch)
+
+    # check pre state
+    assert spec.get_current_epoch(state) < fork_epoch
+
+    yield "pre", state
+
+    # irregular state transition to handle fork:
+    blocks = []
+    # since there are slashed validators, set with_block=False here
+    state, _ = do_fork(state, spec, post_spec, fork_epoch, with_block=False)
+    slashed_indices = [index for index, validator in enumerate(state.validators) if validator.slashed]
+
+    # continue regular state transition with new spec into next epoch
+    transition_to_next_epoch_and_append_blocks(
+        post_spec,
+        state,
+        post_tag,
+        blocks,
+        only_last_block=True,
+        ignoring_proposers=slashed_indices,
+    )
+
+    yield "blocks", blocks
+    yield "post", state
+
+
 @with_fork_metas([ForkMeta(pre_fork_name=pre, post_fork_name=post, fork_epoch=2) for pre, post in ALL_PRE_POST_FORKS])
 def test_transition_missing_first_post_block(state, fork_epoch, spec, post_spec, pre_tag, post_tag):
     """