simulators/eth2/dencun: Tests cases docs

Author: marioevz

simulators/eth2/dencun/suites/builder/TESTS.md

@@ -65,12 +65,12 @@ Test canonical chain includes deneb payloads built by the builder api.
 
 - For each blob transaction on the execution chain, the blob sidecars are available for the beacon block at the same height
 - The beacon block lists the correct commitments for each blob
-- After all other verifications are done, the beacon chain is able to finalize the current epoch
 - Verify that the builder, up to before Deneb fork, has been able to produce blocks and they have been included in the canonical chain
-- Circuit breaker correctly kicks in and disables the builder workflow
-	- Implicitly verified by missed-slot count
-- Builder workflow is re-enabled after the circuit breaker timeout
-- There are no more than 10 missed slots on the latest epoch
+- After Deneb fork, the builder must be able to include blocks with blobs in the canonical chain, which implicitly verifies:
+	- Consensus client is able to properly format header requests to the builder
+	- Consensus client is able to properly format blinded signed requests to the builder
+	- No signed block contained an invalid format or signature
+	- Test fails with a timeout if no payload with blobs is produced after the fork
 
 ### - Deneb Builder Builds Block With Invalid Beacon Root, Correct State Root
 
@@ -116,12 +116,10 @@ building payloads with invalid parent beacon block root.
 
 - For each blob transaction on the execution chain, the blob sidecars are available for the beacon block at the same height
 - The beacon block lists the correct commitments for each blob
-- After all other verifications are done, the beacon chain is able to finalize the current epoch
 - Verify that the builder, up to before Deneb fork, has been able to produce blocks and they have been included in the canonical chain
-- Circuit breaker correctly kicks in and disables the builder workflow
-	- Implicitly verified by missed-slot count
-- Builder workflow is re-enabled after the circuit breaker timeout
-- There are no more than 10 missed slots on the latest epoch
+- After Deneb fork, the builder starts producing invalid payloads, verify that:
+	- None of the produced payloads are included in the canonical chain
+- Since action causes missed slot, verify that the circuit breaker correctly kicks in and disables the builder workflow. Builder starts corrupting payloads after fork, hence a single block in the canonical chain after the fork is enough to verify the circuit breaker
 
 ### - Deneb Builder Errors Out on Header Requests After Deneb Transition
 
@@ -167,13 +165,9 @@ returning error on header request after deneb transition.
 
 - For each blob transaction on the execution chain, the blob sidecars are available for the beacon block at the same height
 - The beacon block lists the correct commitments for each blob
-- After all other verifications are done, the beacon chain is able to finalize the current epoch
 - Verify that the builder, up to before Deneb fork, has been able to produce blocks and they have been included in the canonical chain
-- After Deneb fork, the builder must be able to include blocks with blobs in the canonical chain, which implicitly verifies:
-	- Consensus client is able to properly format header requests to the builder
-	- Consensus client is able to properly format blinded signed requests to the builder
-	- No signed block contained an invalid format or signature
-- There are no more than 1 missed slots on the latest epoch
+- After Deneb fork, the builder starts producing invalid payloads, verify that:
+	- None of the produced payloads are included in the canonical chain
 
 ### - Deneb Builder Errors Out on Signed Blinded Beacon Block/Blob Sidecars Submission After Deneb Transition
 
@@ -219,12 +213,10 @@ returning error on unblinded payload request after deneb transition.
 
 - For each blob transaction on the execution chain, the blob sidecars are available for the beacon block at the same height
 - The beacon block lists the correct commitments for each blob
-- After all other verifications are done, the beacon chain is able to finalize the current epoch
 - Verify that the builder, up to before Deneb fork, has been able to produce blocks and they have been included in the canonical chain
-- Circuit breaker correctly kicks in and disables the builder workflow
-	- Implicitly verified by missed-slot count
-- Builder workflow is re-enabled after the circuit breaker timeout
-- There are no more than 10 missed slots on the latest epoch
+- After Deneb fork, the builder starts producing invalid payloads, verify that:
+	- None of the produced payloads are included in the canonical chain
+- Since action causes missed slot, verify that the circuit breaker correctly kicks in and disables the builder workflow. Builder starts corrupting payloads after fork, hence a single block in the canonical chain after the fork is enough to verify the circuit breaker
 
 ### - Deneb Builder Builds Block With Invalid Payload Version
 
@@ -270,13 +262,9 @@ version is outdated (capella instead of deneb).
 
 - For each blob transaction on the execution chain, the blob sidecars are available for the beacon block at the same height
 - The beacon block lists the correct commitments for each blob
-- After all other verifications are done, the beacon chain is able to finalize the current epoch
 - Verify that the builder, up to before Deneb fork, has been able to produce blocks and they have been included in the canonical chain
-- After Deneb fork, the builder must be able to include blocks with blobs in the canonical chain, which implicitly verifies:
-	- Consensus client is able to properly format header requests to the builder
-	- Consensus client is able to properly format blinded signed requests to the builder
-	- No signed block contained an invalid format or signature
-- There are no more than 1 missed slots on the latest epoch
+- After Deneb fork, the builder starts producing invalid payloads, verify that:
+	- None of the produced payloads are included in the canonical chain
 
 ### - Deneb Builder Builds Block With Invalid Beacon Root, Incorrect State Root
 
@@ -327,10 +315,8 @@ produced locally and results in an empty slot.
 
 - For each blob transaction on the execution chain, the blob sidecars are available for the beacon block at the same height
 - The beacon block lists the correct commitments for each blob
-- After all other verifications are done, the beacon chain is able to finalize the current epoch
 - Verify that the builder, up to before Deneb fork, has been able to produce blocks and they have been included in the canonical chain
-- Circuit breaker correctly kicks in and disables the builder workflow
-	- Implicitly verified by missed-slot count
-- Builder workflow is re-enabled after the circuit breaker timeout
-- There are no more than 10 missed slots on the latest epoch
+- After Deneb fork, the builder starts producing invalid payloads, verify that:
+	- None of the produced payloads are included in the canonical chain
+- Since action causes missed slot, verify that the circuit breaker correctly kicks in and disables the builder workflow. Builder starts corrupting payloads after fork, hence a single block in the canonical chain after the fork is enough to verify the circuit breaker
 

simulators/eth2/dencun/suites/builder/config.go

@@ -1,7 +1,6 @@
 package suite_builder
 
 import (
-	"fmt"
 	"math/big"
 
 	"github.com/ethereum/hive/simulators/eth2/common/clients"
@@ -110,19 +109,21 @@ func (ts BuilderTestSpec) GetDescription() *utils.Description {
 	- Nodes have the mock-builder configured as builder endpoint`)
 	desc.Add(utils.CategoryVerificationsConsensusClient, `
 	- Verify that the builder, up to before Deneb fork, has been able to produce blocks and they have been included in the canonical chain`)
-	if ts.InvalidPayloadCaughtBeforeReveal() {
-		desc.Add(utils.CategoryVerificationsConsensusClient, fmt.Sprintf(`
+	if ts.BuilderProducesValidPayload() {
+		desc.Add(utils.CategoryVerificationsConsensusClient, `
 	- After Deneb fork, the builder must be able to include blocks with blobs in the canonical chain, which implicitly verifies:
 		- Consensus client is able to properly format header requests to the builder
 		- Consensus client is able to properly format blinded signed requests to the builder
 		- No signed block contained an invalid format or signature
-	- There are no more than %d missed slots on the latest epoch`, MAX_MISSED_SLOTS_NO_CIRCUIT_BREAKER))
+		- Test fails with a timeout if no payload with blobs is produced after the fork`)
 	} else {
-		desc.Add(utils.CategoryVerificationsConsensusClient, fmt.Sprintf(`
-		- Circuit breaker correctly kicks in and disables the builder workflow
-			- Implicitly verified by missed-slot count
-		- Builder workflow is re-enabled after the circuit breaker timeout
-		- There are no more than %d missed slots on the latest epoch`, MAX_MISSED_SLOTS_BEFORE_CIRCUIT_BREAKER))
+		desc.Add(utils.CategoryVerificationsConsensusClient, `
+	- After Deneb fork, the builder starts producing invalid payloads, verify that:
+		- None of the produced payloads are included in the canonical chain`)
+		if ts.CausesMissedSlot() {
+			desc.Add(utils.CategoryVerificationsConsensusClient, `
+		- Since action causes missed slot, verify that the circuit breaker correctly kicks in and disables the builder workflow. Builder starts corrupting payloads after fork, hence a single block in the canonical chain after the fork is enough to verify the circuit breaker`)
+		}
 	}
 	return desc
 }

simulators/eth2/dencun/suites/builder/execution.go

@@ -32,7 +32,7 @@ func (ts BuilderTestSpec) WaitForFork(
 	config *tn.Config,
 ) {
 	extraSlotsTimeout := beacon.Slot(4)
-	if !ts.InvalidPayloadCaughtBeforeReveal() {
+	if ts.CausesMissedSlot() {
 		// The error cannot be caught by the consensus client until the payload is revealed,
 		// and the beacon block had been signed. Therefore there will be missed slots
 		// until the circuit breaker kicks in.
@@ -83,29 +83,46 @@ func (ts BuilderTestSpec) ExecutePostFork(
 	}
 }
 
-func (ts BuilderTestSpec) InvalidPayloadCaughtBeforeReveal() bool {
-	// If true, the invalid payload should be caught by the consensus client and not be signed.
-	// The consensus client should fallback to local block production and not miss any slots.
-	if ts.ErrorOnHeaderRequest {
+func (ts BuilderTestSpec) BuilderProducesValidPayload() bool {
+	// If true, the builder is expected to produce a valid payload that should
+	// be included in the canonical chain.
+	if ts.ErrorOnHeaderRequest || ts.InvalidPayloadVersion ||
+		ts.ErrorOnPayloadReveal || ts.InvalidatePayload != "" ||
+		ts.InvalidatePayloadAttributes != "" {
+		return false
+	}
+	return true
+}
+
+func (ts BuilderTestSpec) CausesMissedSlot() bool {
+	// If true, the test is expected to cause a missed slot.
+	if ts.ErrorOnHeaderRequest || ts.InvalidPayloadVersion {
 		// An error on header request to the builder should fallback to local block production,
-		// and hence there isn't even a blinded payload to process.
+		// hence no missed slot.
+		return false
+	}
+	if ts.ErrorOnPayloadReveal {
+		// An error on payload reveal means that, since the mock builder does _not_ relay unblinded
+		// payloads to the p2p network, the consensus client will not be able to produce a block
+		// for the slot, and hence a missed slot is expected.
 		return true
 	}
-	if ts.InvalidPayloadVersion {
-		// An invalid payload version should be caught by the consensus client
-		// when the built blinded payload is received.
+	if ts.InvalidatePayloadAttributes == "" && ts.InvalidatePayload == "" {
+		// If no payload is invalidated, then no missed slot is expected.
+		return false
+	}
+	if ts.InvalidatePayload != "" {
+		// An invalid payload cannot be detected by the consensus client until the payload is
+		// revealed, and the beacon block had been signed. Therefore there will be missed slots.
 		return true
 	}
-	if ts.InvalidatePayloadAttributes != "" {
-		// An invalid payload attribute modification should be caught by the consensus client
-		// when the built blinded payload is received most of the times.
-		if ts.InvalidatePayloadAttributes != mock_builder.INVALIDATE_ATTR_BEACON_ROOT {
-			// Invalid Beacon Root is special because the modified value is not in the response,
-			// and hence can only be detected by the consensus client after the payload is
-			// revealed and executed.
-			return true
-		}
+	if ts.InvalidatePayloadAttributes == mock_builder.INVALIDATE_ATTR_BEACON_ROOT {
+		// Invalid Beacon Root is special because the modified value is not in the response,
+		// and hence can only be detected by the consensus client after the payload is
+		// revealed and executed.
+		return true
 	}
+	// The rest of payload attributes invalidations can be detected by the consensus client
 	return false
 }
 
@@ -251,7 +268,7 @@ func (ts BuilderTestSpec) Verify(
 			}
 
 			var max_missed_slots uint64 = 0
-			if ts.InvalidPayloadCaughtBeforeReveal() {
+			if !ts.CausesMissedSlot() {
 				// These errors should be caught by the CL client when the built blinded
 				// payload is received. Hence, a low number of missed slots is expected.
 				max_missed_slots = MAX_MISSED_SLOTS_NO_CIRCUIT_BREAKER

simulators/eth2/dencun/suites/p2p/gossip/blobs/TESTS.md

@@ -259,7 +259,7 @@ delay makes the test more deterministic.
 #### Blobber Behavior
 
 
-- Create an equivocating block by modifying the graffiti
+- Create an invalid equivocating block by modifying the graffiti
 - Sign both blocks
 - Generate the sidecars out of the correct block only
 - Broadcast the blob sidecars
@@ -357,7 +357,7 @@ Test chain health if a proposer sends equivocating blob sidecars (equivocating b
 #### Blobber Behavior
 
 
-- Create an equivocating block by modifying the graffiti
+- Create an invalid equivocating block by modifying the graffiti
 - Sign both blocks
 - Generate the sidecars out of the equivocating signed block only
 - Broadcast the original signed block only
@@ -406,7 +406,7 @@ Test chain health if a proposer sends equivocating blob sidecars (equivocating b
 #### Blobber Behavior
 
 
-- Create an equivocating block by modifying the graffiti
+- Create an invalid equivocating block by modifying the graffiti
 - Sign both blocks
 - Generate the sidecars out of the equivocating signed block only
 - Broadcast the blob sidecars with the equivocating block header