Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PeerDAS #1

Draft
wants to merge 74 commits into
base: temp-das-start
Choose a base branch
from
Draft

PeerDAS #1

wants to merge 74 commits into from

Commits on Apr 9, 2024

  1. Skip EL part of graffiti when EL info not available (Consensys#8175)

    * Skip EL part of graffiti when EL info not available (previously was NA0000)
    
    * Change EL tracker to push either UNKNOWN (when failed) or valuable version
    zilm13 authored Apr 9, 2024
    Configuration menu
    Copy the full SHA
    78911c1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3172d2f View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    4603466 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    bf17838 View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2024

  1. Configuration menu
    Copy the full SHA
    6a5b007 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2c2249a View commit details
    Browse the repository at this point in the history
  3. Implement new KZG methods

    Nashatyrev committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    008ab85 View commit details
    Browse the repository at this point in the history
  4. Electra spec config and builder (Consensys#8183)

    * Electra spec config and builder
    
    There were fields missing for EIP-7251, and also some of the attestation fields. I added the entire diff from consensus-specs while I was going, it seemed easiest...
    
    Signed-off-by: Paul Harris <paul.harris@consensys.net>
    rolfyone authored Apr 10, 2024
    Configuration menu
    Copy the full SHA
    91b4e3d View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2024

  1. Added a timeout to state regeneration (Consensys#8097)

    Added a development flag to allow us to set state regeneration timeout, with a 120 second timeout by default, and a tiny bit of sanity around the flag not being less than 1.
    
    Signed-off-by: Paul Harris <paul.harris@consensys.net>
    rolfyone authored Apr 11, 2024
    Configuration menu
    Copy the full SHA
    514539b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c4b9a7a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    ad5e63f View commit details
    Browse the repository at this point in the history
  4. Electra MaxEB state changes (Consensys#8186)

    Signed-off-by: Paul Harris <paul.harris@consensys.net>
    rolfyone authored Apr 11, 2024
    Configuration menu
    Copy the full SHA
    8ac4353 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2024

  1. Add DAS Ssz data structures and config constants (Consensys#12)

    * Add KZG prefixes to all struct in the 'kzg' package to avoid further ambiguity
    * Add DAS Ssz data structures and config constants
    Nashatyrev authored Apr 12, 2024
    Configuration menu
    Copy the full SHA
    220d422 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    139edad View commit details
    Browse the repository at this point in the history
  3. Merge remote-tracking branch 'refs/remotes/Consensys/master' into das

    # Conflicts:
    #	ethereum/spec/src/main/java/tech/pegasys/teku/spec/config/SpecConfigElectraImpl.java
    #	ethereum/spec/src/main/java/tech/pegasys/teku/spec/config/builder/ElectraBuilder.java
    #	ethereum/spec/src/main/resources/tech/pegasys/teku/spec/config/presets/mainnet/electra.yaml
    #	ethereum/spec/src/main/resources/tech/pegasys/teku/spec/config/presets/minimal/electra.yaml
    #	ethereum/spec/src/main/resources/tech/pegasys/teku/spec/config/presets/swift/electra.yaml
    #	ethereum/spec/src/test/java/tech/pegasys/teku/spec/config/SpecConfigElectraTest.java
    Nashatyrev committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    45559ca View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    63ff01c View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    8a7a866 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    ce351f5 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    4374c71 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    4cd8af2 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    b37374e View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2024

  1. Add DataColumnSidecarSubnetSubscriptions

    Replace DataColumnSidecarGossipManager with attestation like version
    Nashatyrev committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    ace797f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    25cb4cd View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1350793 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    27ad225 View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2024

  1. Fix copy/paste artifacts

    Nashatyrev committed Apr 16, 2024
    Configuration menu
    Copy the full SHA
    5ea4f8d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    47c94a4 View commit details
    Browse the repository at this point in the history
  3. Gossip (part 2) (Consensys#15)

    * Update GossipForkManager and Eth2P2PNetwork
    * Add CUSTODY_REQUIREMENT constant
    * Create DataColumnSidecarSubnetBackboneSubscriber
    Nashatyrev authored Apr 16, 2024
    Configuration menu
    Copy the full SHA
    b96f8b1 View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2024

  1. spotless apply (Consensys#16)

    zilm13 authored Apr 18, 2024
    Configuration menu
    Copy the full SHA
    6b50aeb View commit details
    Browse the repository at this point in the history
  2. More codestyle fixes

    zilm13 committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    6dc808b View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2024

  1. Merge pull request Consensys#18 from zilm13/more-codestyle

    More codestyle fixes
    Nashatyrev authored Apr 19, 2024
    Configuration menu
    Copy the full SHA
    db62b9a View commit details
    Browse the repository at this point in the history
  2. Central DAS interfaces sketches and some Custody implementation (Cons…

    …ensys#17)
    
    * Add a number of interface sketches
    * Add draft implementations of Custody and CustodySync
    * Add necessary spec config constants and functions
    Nashatyrev authored Apr 19, 2024
    Configuration menu
    Copy the full SHA
    c3a2ca4 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    6c40e3c View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. Peerdas db part2 (Consensys#20)

    * Enable non-canonical sidecars cleanup
    * database part2
    zilm13 authored Apr 22, 2024
    Configuration menu
    Copy the full SHA
    294661b View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2024

  1. Some more DAS core interfaces and some draft implementations (Consens…

    …ys#21)
    
    * DataColumnReqResp to abstract over network Req/Resp implementation
    * DataColumnPeerSearcher to abstract over ConnectionManager + PeerSelectionStrategy
    * DataColumnManager which
            notifies on peer connect/disconnect
            can ban peers returning invalid req/resp responses (this could probably fit into existing peers scoring classes)
    * DataAvailabilitySampler: minimalistic for now
    * Add simple implementations ofDataColumnSidecarRetriever and DataAvailabilitySampler interfaces
    Nashatyrev authored Apr 23, 2024
    Configuration menu
    Copy the full SHA
    ad67fca View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2024

  1. DataColumnSidecars database tests (Consensys#22)

    * database integration tests
    * found a bug with missed commit on pruning, fixed
    * minor old interface clarification input name
    zilm13 authored Apr 24, 2024
    Configuration menu
    Copy the full SHA
    67b4d56 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2024

  1. Configuration menu
    Copy the full SHA
    1d1f478 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2024

  1. Configuration menu
    Copy the full SHA
    a5b32f3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6886173 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. Remove Electra stuff (Consensys#26)

    * removed all Electra features
    * renamed from Electra to EIP7594, compatible with spec config
    * configs updated to be close to those in spec
    * fixed bug with kzg_commitment_inclusion_proof size
    * fixed all tests
    * modified acceptance test BlockProposalAcceptanceTest to ensure we could run up to EIP7594 fork
    zilm13 authored Apr 30, 2024
    Configuration menu
    Copy the full SHA
    6444f9c View commit details
    Browse the repository at this point in the history
  2. Update DataColumnSidecar kzg verification + proof verification (Conse…

    …nsys#27)
    
    * added structure checks in kzg verification from the spec
    * added proof verification
    * fixed kzg_commitment_proof -> kzg_commitments_proof
    * logging fixes
    * small changes to naming to distinguish with blobs
    zilm13 authored Apr 30, 2024
    Configuration menu
    Copy the full SHA
    f28054f View commit details
    Browse the repository at this point in the history

Commits on May 1, 2024

  1. Merge pull request Consensys#28

    * Fix el methods for EIP7594
    zilm13 authored May 1, 2024
    Configuration menu
    Copy the full SHA
    29eb79f View commit details
    Browse the repository at this point in the history
  2. Merge pull request Consensys#29

    * constructDataColumnSidecars miscHelpers method
    zilm13 authored May 1, 2024
    Configuration menu
    Copy the full SHA
    b3e8079 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2a12f22 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2024

  1. Fix the DataColumnSidecars construction (Consensys#32)

    * Adjust DataColumnSidecar creation
    * Fix the test compilation
    Nashatyrev authored May 3, 2024
    Configuration menu
    Copy the full SHA
    3d6e1fe View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2cf41a1 View commit details
    Browse the repository at this point in the history
  3. Disable BlobSidecars availability check in the post Deneb milestones (C…

    …onsensys#34)
    
    * Disable BlobSidecars availability check in the post Deneb milestones
    * More accurate logging
    Nashatyrev authored May 3, 2024
    Configuration menu
    Copy the full SHA
    7097eab View commit details
    Browse the repository at this point in the history
  4. Integrate DataColumnSidecarCustody for storing gossip inbound/outboun…

    …d sidecars (Consensys#35)
    
    * Integrate DataColumnSidecarCustody for storing gossip inbound/outbound sidecars
    * Add suitable sidecars only in DataColumnSidecarCustodyImpl
    * Change initialization sequence
    * Rename to BlockRootResolver
    * Remove stinky comment :)
    * Skip initing DasCustody if milestone is not supported
    Nashatyrev authored May 3, 2024
    Configuration menu
    Copy the full SHA
    3ac6cc5 View commit details
    Browse the repository at this point in the history

Commits on May 7, 2024

  1. Reference tests for EIP-7594 + utils implementation (Consensys#36)

    * eference tests 1.5.0.alpha.1 with EIP-7594 coverage activated
    * computeCustodyColumnIndexes implemented
    * verifyCellProofBatch implemented, it fails badly on some tests, so it's disabled. jc-kzg update should help I think
    zilm13 authored May 7, 2024
    Configuration menu
    Copy the full SHA
    3c38ebc View commit details
    Browse the repository at this point in the history
  2. Ref tests 1.5.0.alpha.2 + jc-kzg usage update (Consensys#37)

    * Bump reference tests to 1.5.0.alpha.2 + jc-kzg usage update
    * fix old import
    zilm13 authored May 7, 2024
    Configuration menu
    Copy the full SHA
    dedcd96 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. Expose discovery NodeId to DiscoveryPeer and LibP2PPeer (Consensys#38)

    * Add DiscoveryPeer.getNodeId()
    * Use the correct NodeId (from Discovery) instead of wrong Libp2p PeerId when calculating DAS subnets
    * Add Eth2Peer.getDiscoveryNodeId() method.
    Nashatyrev authored May 9, 2024
    Configuration menu
    Copy the full SHA
    8a76b25 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    9e71e72 View commit details
    Browse the repository at this point in the history
  3. DasPeerManagerImpl and DAS req/resp initial drafts (Consensys#40)

    * Refactor:
      *  Make DasPeerCustodyCountSupplier a separate thing
      *  Move DataColumnSidecarRetriever interface to subpackage
    * Add DataColumnPeerManagerImpl
    * Add BatchDataColumnReqResp and implement it in DataColumnPeerManagerImpl
    * Make DataColumnReqResp implementation on top of BatchDataColumnReqResp
    Nashatyrev authored May 9, 2024
    Configuration menu
    Copy the full SHA
    a2c5ac4 View commit details
    Browse the repository at this point in the history
  4. First efforts to launch DasCustodySync (Consensys#41)

    * Add draft init code to create DasCustodySync in BeaconChainController
    * Make the primitive triggering of SimpleSidecarRetriever: every 1 sec
    * Fix GossipForkManager to subscribe currentDataColumnSidecarSubnets on startSubscriptions
    * Use the right NodeId in the DataColumnPeerManagerImpl
    * BeaconChainController.initDasCustody():
      * subscribe missing elements to slot channel events.
      * Add DasCustodySync start initiator
    Nashatyrev authored May 9, 2024
    Configuration menu
    Copy the full SHA
    23f1416 View commit details
    Browse the repository at this point in the history
  5. Fixes to make sync working (Consensys#42)

    * Fix NodeRecordConverter.convertPublicKeyToNodeId() (would need more api from discovery to make it more or less nice)
    * Align computeDataColumnSidecarBackboneSubnets with current spec
    * Make DataColumnBlockRootResolver resolve only blocks with non-empty blob list: blocks without blobs have no columns so we are just ignoring them in the DAS subsystem
    * Relax DataColumnSidecarsByRoot RPC slot range requirement for now: will return them back when adopt byRange method variant
    * Fix bug in SimpleSidecarRetriever
    * Fix something in DasCustodySync
    * Fix updating DB firstIncompleteSlot
    Nashatyrev authored May 9, 2024
    Configuration menu
    Copy the full SHA
    b7ae7b9 View commit details
    Browse the repository at this point in the history
  6. Spotless (Consensys#43)

    * spotless
    * fix compiler warns
    * make tests compile
    Nashatyrev authored May 9, 2024
    Configuration menu
    Copy the full SHA
    1575fda View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. Track extra subnet count by gossip subscriptions (Consensys#44)

    * Implement GossipTopicDasPeerCustodyTracker
    * Refactor DataColumnBlockRootResolver to move more logic from BeaconChainController to DataColumnSidecarCustodyImpl
    * Fix the case when extra_column_subnets too large
    Nashatyrev authored May 10, 2024
    Configuration menu
    Copy the full SHA
    8696e58 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2024

  1. Configuration menu
    Copy the full SHA
    0e64217 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. Read numberOfColumns from config instead of calculating (Consensys#46)

    * NUMBER_OF_COLUMNS is now in config, better to read it
    * removed hardcode from fieldElementsPerCell as there is no more division by it
    zilm13 authored May 14, 2024
    Configuration menu
    Copy the full SHA
    adfb089 View commit details
    Browse the repository at this point in the history
  2. Reduce RPC column request limit (Consensys#47)

    * Reduce MAX_REQUEST_DATA_COLUMN_SIDECARS const to 512
    * Fix propagation of call exception to returned future
    * Fix/refactor SimpleSidecarRetriever to respect maxRequestCount and the number of currently running requests per peer
    * Fix compile error from prev PR
    Nashatyrev committed May 14, 2024
    Configuration menu
    Copy the full SHA
    63ab9aa View commit details
    Browse the repository at this point in the history
  3. Add some logs

    Nashatyrev committed May 14, 2024
    Configuration menu
    Copy the full SHA
    48edab4 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    fc23e9f View commit details
    Browse the repository at this point in the history
  5. Fix compiler warning

    Nashatyrev committed May 14, 2024
    Configuration menu
    Copy the full SHA
    1aca86c View commit details
    Browse the repository at this point in the history
  6. More intensive DAS logging

    Nashatyrev committed May 14, 2024
    Configuration menu
    Copy the full SHA
    f9d21fb View commit details
    Browse the repository at this point in the history

Commits on May 15, 2024

  1. Configuration menu
    Copy the full SHA
    a2731ab View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7dcf5b3 View commit details
    Browse the repository at this point in the history
  3. Fix: 'complete' slot in DAS DB should be finalized (Consensys#50)

    * FirstIncompleteSlot: 'complete' slot should be finalized
    * Add logging of inbound DAS RPC requests
    * Add more logging details to DataColumnSidecarDBImpl
    * Add '[nyota]' prefix to all DAS related logs
    Nashatyrev authored May 15, 2024
    Configuration menu
    Copy the full SHA
    7ff3722 View commit details
    Browse the repository at this point in the history
  4. das-pr/3-peers-milestone (Consensys#52)

    * Switch to 'develop' libp2p version
    * Add logs on peer connect/disconnect
    * Log: fix nodeId output
    Nashatyrev authored May 15, 2024
    Configuration menu
    Copy the full SHA
    94b147a View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. Better logging for disconnection reasons (Consensys#53)

    * Better logging for disconnection reasons 
    * request limit increase
    zilm13 authored May 16, 2024
    Configuration menu
    Copy the full SHA
    7d4d65a View commit details
    Browse the repository at this point in the history
  2. Fix remote subnet tracking, other stuff (Consensys#54)

    * SimpleSidecarRetriever: Add more synchronized
    * SimpleSidecarRetriever: add logging stats on the number of custody peers for columns
    * P2PConfig.getDasExtraCustodySubnetCount: capped in place with Spec
    * GossipTopicDasPeerCustodyTracker: track subscriptions periodically since right after cionnection it may fail as a remote peer didn't yest subscribed
    * Cleanup totl/extraCustodySubnetCount mess
    Nashatyrev authored May 16, 2024
    Configuration menu
    Copy the full SHA
    0625472 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    9fd3f7b View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    4abba1f View commit details
    Browse the repository at this point in the history

Commits on May 17, 2024

  1. Implement long polling req/resp getDataColumnByRoot() method (Consens…

    …ys#57)
    
    * Implement long polling req/resp getDataColumnByRoot() method
    * Add DataColumnIdentifier.createFromSidecar() factory method
    Nashatyrev authored May 17, 2024
    Configuration menu
    Copy the full SHA
    bc33599 View commit details
    Browse the repository at this point in the history
  2. Build fix + minor name fixes

    zilm13 committed May 17, 2024
    Configuration menu
    Copy the full SHA
    0295ebd View commit details
    Browse the repository at this point in the history