Various fixes to custody range sync #6004

jimmygchen · 2024-06-27T00:36:29Z

Issue Addressed

The current behaviour is the syncing node will start sending batches on startup, and if there isn't any peers on custody column subnets, the DataColumnByRange requests will return an error immediately, triggering another retry. And this may cause the chain to be removed and peers to disconnect shortly.

This PR addresses the above and a few other sync related issues:

Only request batches when there are good peers across all custody subnets.
Use subscribed peers on subnets before MetaDataV3 is implemented as it's difficult to get custody peers on small devnets without identifying supernodes.
Remove peer_id matching when injecting error because multiple peers are used for range requests. This prevents sync from getting stuck.
Use randomized custodial peer to avoid repeatedly sending requests to failing peers.
Batch by range request where possible

…stody columns to avoid spaming block requests.

…all custody subnets (sigp#6004) Squashed commit of the following: commit edc584a Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 10:32:19 2024 +1000 Only start requesting batches when there are good peers across all custody columns to avoid spaming block requests.

…ent state.

…all custody subnets (sigp#6004) Squashed commit of the following: commit 9f82497 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 12:57:06 2024 +1000 Add custody peer check before mutating `BatchInfo` to avoid inconsistent state. commit edc584a Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 10:32:19 2024 +1000 Only start requesting batches when there are good peers across all custody columns to avoid spaming block requests.

…all custody subnets (sigp#6004) Squashed commit of the following: commit 0d97026 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 13:51:33 2024 +1000 Add check to cover a case where batch is not processed while waiting for custody peers to become available. commit 9f82497 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 12:57:06 2024 +1000 Add custody peer check before mutating `BatchInfo` to avoid inconsistent state. commit edc584a Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 10:32:19 2024 +1000 Only start requesting batches when there are good peers across all custody columns to avoid spaming block requests.

…for custody peers to become available.

…all custody subnets (sigp#6004) Squashed commit of the following: commit 9f20029 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 14:27:04 2024 +1000 Fix lint and logic error commit 05608b0 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 13:51:33 2024 +1000 Add check to cover a case where batch is not processed while waiting for custody peers to become available. commit 9f82497 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 12:57:06 2024 +1000 Add custody peer check before mutating `BatchInfo` to avoid inconsistent state. commit edc584a Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 10:32:19 2024 +1000 Only start requesting batches when there are good peers across all custody columns to avoid spaming block requests.

…et`.

jimmygchen · 2024-06-27T07:33:53Z

Looks like there are still a few more issues - most likely the issue is that we're assuming enr is always available, will look into that tomorrow.

dapplion

The redundant rotation epoch parameter is also removed (sorry probably should've kept this separate, wasn't expecting it to change many files).

Can you split in a separate PR? The main change is very critical and it can brick sync so best to review on its own

beacon_node/network/src/sync/range_sync/chain.rs

…n all custody subnets (#6004) Squashed commit of the following: commit 129ac92 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Mon Jul 1 12:49:03 2024 +1000 Add metrics and update code comment. commit 03a9dce Author: Jimmy Chen <jchen.tc@gmail.com> Date: Mon Jul 1 12:29:30 2024 +1000 Fall back to default custody requiremnt if peer ENR is not present. commit 4373a28 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Sat Jun 29 01:05:52 2024 +1000 Revert epoch parameter refactor. commit 876ea3b Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 17:21:04 2024 +1000 Add test for `get_custody_peers_for_column` commit de05355 Merge: 4079d2e 7206909 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 16:06:28 2024 +1000 Merge branch 'das' into custody-sync-peers commit 4079d2e Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 15:35:39 2024 +1000 Fix `good_peers_on_subnet` always returning false for `DataColumnSubnet`. commit 9f20029 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 14:27:04 2024 +1000 Fix lint and logic error commit 05608b0 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 13:51:33 2024 +1000 Add check to cover a case where batch is not processed while waiting for custody peers to become available. commit 9f82497 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 12:57:06 2024 +1000 Add custody peer check before mutating `BatchInfo` to avoid inconsistent state. commit edc584a Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 10:32:19 2024 +1000 Only start requesting batches when there are good peers across all custody columns to avoid spaming block requests.

…n all custody subnets (#6004) Squashed commit of the following: commit cfb3ebc Author: Jimmy Chen <jchen.tc@gmail.com> Date: Mon Jul 1 12:49:03 2024 +1000 Add metrics and update code comment. commit 03a9dce Author: Jimmy Chen <jchen.tc@gmail.com> Date: Mon Jul 1 12:29:30 2024 +1000 Fall back to default custody requiremnt if peer ENR is not present. commit 4373a28 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Sat Jun 29 01:05:52 2024 +1000 Revert epoch parameter refactor. commit 876ea3b Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 17:21:04 2024 +1000 Add test for `get_custody_peers_for_column` commit de05355 Merge: 4079d2e 7206909 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 16:06:28 2024 +1000 Merge branch 'das' into custody-sync-peers commit 4079d2e Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 15:35:39 2024 +1000 Fix `good_peers_on_subnet` always returning false for `DataColumnSubnet`. commit 9f20029 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 14:27:04 2024 +1000 Fix lint and logic error commit 05608b0 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 13:51:33 2024 +1000 Add check to cover a case where batch is not processed while waiting for custody peers to become available. commit 9f82497 Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 12:57:06 2024 +1000 Add custody peer check before mutating `BatchInfo` to avoid inconsistent state. commit edc584a Author: Jimmy Chen <jchen.tc@gmail.com> Date: Thu Jun 27 10:32:19 2024 +1000 Only start requesting batches when there are good peers across all custody columns to avoid spaming block requests.

…ve peer_id matching when injecting error because multiple peers are used for range requests. Use randomized custodial peer to avoid repeatedly sending requests to failing peers. Batch by range request where possible.

jimmygchen · 2024-07-08T06:02:48Z

beacon_node/network/src/sync/network_context.rs

+        let mut peer_id_to_request_map = HashMap::new();
+
+        for column_index in custody_indexes {
+            let Some(custody_peer) = self.get_random_custodial_peer(epoch, *column_index) else {


There's potentially an issue with the peer selection approach here - as peer selection was previously done one layer up:

request_batches: here we use a list of "idle peers" and randomly select one

retry_batch_download: here we sort peers prioritizing unrelated peers with less active requests

Are these selection rules important? I've ran into some issues with concurrent requests to the same peer with the approach on this branch, and need to investigate further.

Checked with Age, and it should be fine to send requests to peers that aren't "idle".

The peer selection logic here isn't ideal - we should probably avoid retrying peers that failed to return a response. Perhaps we could pass the sorted peer list into this method?

Added a comment for this, probably requires some thoughts and refactoring as we don't currently track data_column_by_range peers in BatchState.

Note to self: we might also need to apply some of these changes to backfill sync.

# Conflicts: # beacon_node/network/src/sync/network_context.rs # beacon_node/network/src/sync/range_sync/batch.rs

# Conflicts: # beacon_node/network/src/sync/sampling.rs

@jimmygchen

* 1D PeerDAS prototype: Data format and Distribution (#5050) * Build and publish column sidecars. Add stubs for gossip. * Add blob column subnets * Add `BlobColumnSubnetId` and initial compute subnet logic. * Subscribe to blob column subnets. * Introduce `BLOB_COLUMN_SUBNET_COUNT` based on DAS configuration parameter changes. * Fix column sidecar type to use `VariableList` for data. * Fix lint errors. * Update types and naming to latest consensus-spec #3574. * Fix test and some cleanups. * Merge branch 'unstable' into das * Merge branch 'unstable' into das * Merge branch 'unstable' into das # Conflicts: # consensus/types/src/chain_spec.rs * Add `DataColumnSidecarsByRoot ` req/resp protocol (#5196) * Add stub for `DataColumnsByRoot` * Add basic implementation of serving RPC data column from DA checker. * Store data columns in early attester cache and blobs db. * Apply suggestions from code review Co-authored-by: Eitan Seri-Levi <eserilev@gmail.com> Co-authored-by: Jacob Kaufmann <jacobkaufmann18@gmail.com> * Fix build. * Store `DataColumnInfo` in database and various cleanups. * Update `DataColumnSidecar` ssz max size and remove panic code. --------- Co-authored-by: Eitan Seri-Levi <eserilev@gmail.com> Co-authored-by: Jacob Kaufmann <jacobkaufmann18@gmail.com> * feat: add DAS KZG in data col construction (#5210) * feat: add DAS KZG in data col construction * refactor data col sidecar construction * refactor: add data cols to GossipVerifiedBlockContents * Disable windows tests for `das` branch. (c-kzg doesn't build on windows) * Formatting and lint changes only. * refactor: remove iters in construction of data cols * Update vec capacity and error handling. * Add `data_column_sidecar_computation_seconds` metric. --------- Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * Merge branch 'unstable' into das # Conflicts: # .github/workflows/test-suite.yml # beacon_node/lighthouse_network/src/types/topics.rs * fix: update data col subnet count from 64 to 32 (#5413) * feat: add peerdas custody field to ENR (#5409) * feat: add peerdas custody field to ENR * add hash prefix step in subnet computation * refactor test and fix possible u64 overflow * default to min custody value if not present in ENR * Merge branch 'unstable' into das * Merge branch 'unstable' into das-unstable-merge-0415 # Conflicts: # Cargo.lock # beacon_node/beacon_chain/src/data_availability_checker.rs # beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # beacon_node/beacon_chain/src/data_availability_checker/processing_cache.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/network/src/network_beacon_processor/mod.rs # beacon_node/network/src/sync/block_lookups/tests.rs # crypto/kzg/Cargo.toml * Merge remote-tracking branch 'sigp/unstable' into das * Merge remote-tracking branch 'sigp/unstable' into das * Fix merge conflicts. * Send custody data column to `DataAvailabilityChecker` for determining block importability (#5570) * Only import custody data columns after publishing a block. * Add `subscribe-all-data-column-subnets` and pass custody column count to `availability_cache`. * Add custody requirement checks to `availability_cache`. * Fix config not being passed to DAChecker and add more logging. * Introduce `peer_das_epoch` and make blobs and columns mutually exclusive. * Add DA filter for PeerDAS. * Fix data availability check and use test_logger in tests. * Fix subscribe to all data column subnets not working correctly. * Fix tests. * Only publish column sidecars if PeerDAS is activated. Add `PEER_DAS_EPOCH` chain spec serialization. * Remove unused data column index in `OverflowKey`. * Fix column sidecars incorrectly produced when there are no blobs. * Re-instate index to `OverflowKey::DataColumn` and downgrade noisy debug log to `trace`. * DAS sampling on sync (#5616) * Data availability sampling on sync * Address @jimmygchen review * Trigger sampling * Address some review comments and only send `SamplingBlock` sync message after PEER_DAS_EPOCH. --------- Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * Merge branch 'unstable' into das # Conflicts: # Cargo.lock # Cargo.toml # beacon_node/beacon_chain/src/block_verification.rs # beacon_node/http_api/src/publish_blocks.rs # beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs # beacon_node/lighthouse_network/src/rpc/protocol.rs # beacon_node/lighthouse_network/src/types/pubsub.rs # beacon_node/network/src/sync/block_lookups/single_block_lookup.rs # beacon_node/store/src/hot_cold_store.rs # consensus/types/src/beacon_state.rs # consensus/types/src/chain_spec.rs # consensus/types/src/eth_spec.rs * Merge branch 'unstable' into das * Re-process early sampling requests (#5569) * Re-process early sampling requests # Conflicts: # beacon_node/beacon_processor/src/work_reprocessing_queue.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/network/src/network_beacon_processor/rpc_methods.rs * Update beacon_node/beacon_processor/src/work_reprocessing_queue.rs Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * Add missing var * Beta compiler fixes and small typo fixes. * Remove duplicate method. --------- Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * Merge remote-tracking branch 'sigp/unstable' into das * Fix merge conflict. * Add data columns by root to currently supported protocol list (#5678) * Add data columns by root to currently supported protocol list. * Add missing data column by roots handling. * Merge branch 'unstable' into das # Conflicts: # Cargo.lock # Cargo.toml # beacon_node/network/src/sync/block_lookups/tests.rs # beacon_node/network/src/sync/manager.rs * Fix simulator tests on `das` branch (#5731) * Bump genesis delay in sim tests as KZG setup takes longer for DAS. * Fix incorrect YAML spacing. * DataColumnByRange boilerplate (#5353) * add boilerplate * fmt * PeerDAS custody lookup sync (#5684) * Implement custody sync * Lint * Fix tests * Fix rebase issue * Add data column kzg verification and update `c-kzg`. (#5701) * Add data column kzg verification and update `c-kzg`. * Fix incorrect `Cell` size. * Add kzg verification on rpc blocks. * Add kzg verification on rpc data columns. * Rename `PEER_DAS_EPOCH` to `EIP7594_FORK_EPOCH` for client interop. (#5750) * Fetch custody columns in range sync (#5747) * Fetch custody columns in range sync * Clean up todos * Remove `BlobSidecar` construction and publish after PeerDAS activated (#5759) * Avoid building and publishing blob sidecars after PeerDAS. * Ignore gossip blobs with a slot greater than peer das activation epoch. * Only attempt to verify blob count and import blobs before PeerDAS. * #5684 review comments (#5748) * #5684 review comments. * Doc and message update only. * Fix incorrect condition when constructing `RpcBlock` with `DataColumn`s * Make sampling tests deterministic (#5775) * PeerDAS spec tests (#5772) * Add get_custody_columns spec tests. * Add kzg merkle proof spec tests. * Add SSZ spec tests. * Add remaining KZG tests * Load KZG only once per process, exclude electra tests and add missing SSZ tests. * Fix lint and missing changes. * Ignore macOS generated file. * Merge remote branch 'sigp/unstable' into das * Merge remote tracking branch 'origin/unstable' into das * Implement unconditional reconstruction for supernodes (#5781) * Implement unconditional reconstruction for supernodes * Move code into KzgVerifiedCustodyDataColumn * Remove expect * Add test * Thanks justin * Add withhold attack mode for interop (#5788) * Add withhold attack mode * Update readme * Drop added readmes * Undo styling changes * Add column gossip verification and handle unknown parent block (#5783) * Add column gossip verification and handle missing parent for columns. * Review PR * Fix rebase issue * more lint issues :) --------- Co-authored-by: dapplion <35266934+dapplion@users.noreply.github.com> * Trigger sampling on sync events (#5776) * Trigger sampling on sync events * Update beacon_chain.rs * Fix tests * Fix tests * PeerDAS parameter changes for devnet-0 (#5779) * Update PeerDAS parameters to latest values. * Lint fix * Fix lint. * Update hardcoded subnet count to 64 (#5791) * Fix incorrect columns per subnet and config cleanup (#5792) * Tidy up PeerDAS preset and config values. * Fix broken config * Fix DAS branch CI (#5793) * Fix invalid syntax. * Update cli doc. Ignore get_custody_columns test temporarily. * Fix failing test and add verify inclusion test. * Undo accidentally removed code. * Only attempt reconstruct columns once. (#5794) * Re-enable precompute table for peerdas kzg (#5795) * Merge branch 'unstable' into das * Update subscription filter. (#5797) * Remove penalty for duplicate columns (expected due to reconstruction) (#5798) * Revert DAS config for interop testing. Optimise get_custody_columns function. (#5799) * Don't perform reconstruction for proposer node as it already has all the columns. (#5806) * Multithread compute_cells_and_proofs (#5805) * Multi-thread reconstruct data columns * Multi-thread path for block production * Merge branch 'unstable' into das # Conflicts: # .github/workflows/test-suite.yml # beacon_node/network/src/sync/block_lookups/mod.rs # beacon_node/network/src/sync/block_lookups/single_block_lookup.rs # beacon_node/network/src/sync/network_context.rs * Fix CI errors. * Move PeerDAS type-level config to configurable `ChainSpec` (#5828) * Move PeerDAS type level config to `ChainSpec`. * Fix tests * Misc custody lookup improvements (#5821) * Improve custody requests * Type DataColumnsByRootRequestId * Prioritize peers and load balance * Update tests * Address PR review * Merge branch 'unstable' into das * Rename deploy_block in network config (`das` branch) (#5852) * Rename deploy_block.txt to deposit_contract_block.txt * fmt --------- Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com> * Merge branch 'unstable' into das * Fix CI and merge issues. * Merge branch 'unstable' into das # Conflicts: # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # lcli/src/main.rs * Store data columns individually in store and caches (#5890) * Store data columns individually in store and caches * Implement data column pruning * Merge branch 'unstable' into das # Conflicts: # Cargo.lock * Update reconstruction benches to newer criterion version. (#5949) * Merge branch 'unstable' into das # Conflicts: # .github/workflows/test-suite.yml * chore: add `recover_cells_and_compute_proofs` method (#5938) * chore: add recover_cells_and_compute_proofs method * Introduce type alias `CellsAndKzgProofs` to address type complexity. --------- Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * Update `csc` format in ENR and spec tests for devnet-1 (#5966) * Update `csc` format in ENR. * Add spec tests for `recover_cells_and_kzg_proofs`. * Add tests for ENR. * Fix failing tests. * Add protection against invalid csc value in ENR. * Fix lint * Fix csc encoding and decoding (#5997) * Fix data column rpc request not being sent due to incorrect limits set. (#6000) * Fix incorrect inbound request count causing rate limiting. (#6025) * Merge branch 'stable' into das # Conflicts: # beacon_node/network/src/sync/block_lookups/tests.rs # beacon_node/network/src/sync/block_sidecar_coupling.rs # beacon_node/network/src/sync/manager.rs # beacon_node/network/src/sync/network_context.rs # beacon_node/network/src/sync/network_context/requests.rs * Merge remote-tracking branch 'unstable' into das * Add kurtosis config for DAS testing (#5968) * Add kurtosis config for DAS testing. * Fix invalid yaml file * Update network parameter files. * chore: add rust PeerdasKZG crypto library for peerdas functionality and rollback c-kzg dependency to 4844 version (#5941) * chore: add recover_cells_and_compute_proofs method * chore: add rust peerdas crypto library * chore: integrate peerdaskzg rust library into kzg crate * chore(multi): - update `ssz_cell_to_crypto_cell` - update conversion from the crypto cell type to a Vec<u8>. Since the Rust library defines them as references to an array, the conversion is simply `to_vec` * chore(multi): - update rest of code to handle the new crypto `Cell` type - update test case code to no longer use the Box type * chore: cleanup of superfluous conversions * chore: revert c-kzg dependency back to v1 * chore: move dependency into correct order * chore: update rust dependency - This version includes a new method `PeerDasContext::with_num_threads` * chore: remove Default initialization of PeerDasContext and explicitly set the parameters in `new_from_trusted_setup` * chore: cleanup exports * chore: commit updated cargo.lock * Update Cargo.toml Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * chore: rename dependency * chore: update peerdas lib - sets the blst version to 0.3 so that it matches whatever lighthouse is using. Although 0.3.12 is latest, lighthouse is pinned to 0.3.3 * chore: fix clippy lifetime - Rust doesn't allow you to elide the lifetime on type aliases * chore: cargo clippy fix * chore: cargo fmt * chore: update lib to add redundant checks (these will be removed in consensus-specs PR 3819) * chore: update dependency to ignore proofs * chore: update peerdas lib to latest * update lib * chore: remove empty proof parameter --------- Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * Update PeerDAS interop testnet config (#6069) * Update interop testnet config. * Fix typo and remove target peers * Avoid retrying same sampling peer that previously failed. (#6084) * Various fixes to custody range sync (#6004) * Only start requesting batches when there are good peers across all custody columns to avoid spaming block requests. * Add custody peer check before mutating `BatchInfo` to avoid inconsistent state. * Add check to cover a case where batch is not processed while waiting for custody peers to become available. * Fix lint and logic error * Fix `good_peers_on_subnet` always returning false for `DataColumnSubnet`. * Add test for `get_custody_peers_for_column` * Revert epoch parameter refactor. * Fall back to default custody requiremnt if peer ENR is not present. * Add metrics and update code comment. * Add more debug logs. * Use subscribed peers on subnet before MetaDataV3 is implemented. Remove peer_id matching when injecting error because multiple peers are used for range requests. Use randomized custodial peer to avoid repeatedly sending requests to failing peers. Batch by range request where possible. * Remove unused code and update docs. * Add comment * chore: update peerdas-kzg library (#6118) * chore: update peerDAS lib * chore: update library * chore: update library to version that include "init context" benchmarks and optional validation checks * chore: (can remove) -- Add benchmarks for init context * Prevent continuous searchers for low-peer networks (#6162) * Merge branch 'unstable' into das * Fix merge conflicts * Add cli flag to enable sampling and disable by default. (#6209) * chore: Use reference to an array representing a blob instead of an owned KzgBlob (#6179) * add KzgBlobRef type * modify code to use KzgBlobRef * clippy * Remove Deneb blob related changes to maintain compatibility with `c-kzg-4844`. --------- Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * Store computed custody subnets in PeerDB and fix custody lookup test (#6218) * Fix failing custody lookup tests. * Store custody subnets in PeerDB, fix custody lookup test and refactor some methods. * Merge branch 'unstable' into das # Conflicts: # beacon_node/beacon_chain/src/beacon_chain.rs # beacon_node/beacon_chain/src/block_verification_types.rs # beacon_node/beacon_chain/src/builder.rs # beacon_node/beacon_chain/src/data_availability_checker.rs # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # beacon_node/beacon_chain/src/data_column_verification.rs # beacon_node/beacon_chain/src/early_attester_cache.rs # beacon_node/beacon_chain/src/historical_blocks.rs # beacon_node/beacon_chain/tests/store_tests.rs # beacon_node/lighthouse_network/src/discovery/enr.rs # beacon_node/network/src/service.rs # beacon_node/src/cli.rs # beacon_node/store/src/hot_cold_store.rs # beacon_node/store/src/lib.rs # lcli/src/generate_bootnode_enr.rs * Fix CI failures after merge. * Batch sampling requests by peer (#6256) * Batch sampling requests by peer * Fix clippy errors * Fix tests * Add column_index to error message for ease of tracing * Remove outdated comment * Fix range sync never evaluating request as finished, causing it to get stuck. (#6276) * Merge branch 'unstable' into das-0821-merge # Conflicts: # Cargo.lock # Cargo.toml # beacon_node/beacon_chain/src/beacon_chain.rs # beacon_node/beacon_chain/src/data_availability_checker.rs # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # beacon_node/beacon_chain/src/data_column_verification.rs # beacon_node/beacon_chain/src/kzg_utils.rs # beacon_node/beacon_chain/src/metrics.rs # beacon_node/beacon_processor/src/lib.rs # beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs # beacon_node/lighthouse_network/src/rpc/config.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/lighthouse_network/src/rpc/outbound.rs # beacon_node/lighthouse_network/src/rpc/rate_limiter.rs # beacon_node/lighthouse_network/src/service/api_types.rs # beacon_node/lighthouse_network/src/types/globals.rs # beacon_node/network/src/network_beacon_processor/mod.rs # beacon_node/network/src/network_beacon_processor/rpc_methods.rs # beacon_node/network/src/network_beacon_processor/sync_methods.rs # beacon_node/network/src/sync/block_lookups/common.rs # beacon_node/network/src/sync/block_lookups/mod.rs # beacon_node/network/src/sync/block_lookups/single_block_lookup.rs # beacon_node/network/src/sync/block_lookups/tests.rs # beacon_node/network/src/sync/manager.rs # beacon_node/network/src/sync/network_context.rs # consensus/types/src/data_column_sidecar.rs # crypto/kzg/Cargo.toml # crypto/kzg/benches/benchmark.rs # crypto/kzg/src/lib.rs * Fix custody tests and load PeerDAS KZG instead. * Fix ef tests and bench compilation. * Fix failing sampling test. * Merge pull request #6287 from jimmygchen/das-0821-merge Merge `unstable` into `das` 20240821 * Remove get_block_import_status * Merge branch 'unstable' into das * Re-enable Windows release tests. * Address some review comments. * Address more review comments and cleanups. * Comment out peer DAS KZG EF tests for now * Address more review comments and fix build. * Merge branch 'das' of github.com:sigp/lighthouse into das * Unignore Electra tests * Fix metric name * Address some of Pawan's review comments * Merge remote-tracking branch 'origin/unstable' into das * Update PeerDAS network parameters for peerdas-devnet-2 (#6290) * update subnet count & custody req * das network params * update ef tests --------- Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

@jimmygchen

* 1D PeerDAS prototype: Data format and Distribution (sigp#5050) * Build and publish column sidecars. Add stubs for gossip. * Add blob column subnets * Add `BlobColumnSubnetId` and initial compute subnet logic. * Subscribe to blob column subnets. * Introduce `BLOB_COLUMN_SUBNET_COUNT` based on DAS configuration parameter changes. * Fix column sidecar type to use `VariableList` for data. * Fix lint errors. * Update types and naming to latest consensus-spec sigp#3574. * Fix test and some cleanups. * Merge branch 'unstable' into das * Merge branch 'unstable' into das * Merge branch 'unstable' into das # Conflicts: # consensus/types/src/chain_spec.rs * Add `DataColumnSidecarsByRoot ` req/resp protocol (sigp#5196) * Add stub for `DataColumnsByRoot` * Add basic implementation of serving RPC data column from DA checker. * Store data columns in early attester cache and blobs db. * Apply suggestions from code review Co-authored-by: Eitan Seri-Levi <eserilev@gmail.com> Co-authored-by: Jacob Kaufmann <jacobkaufmann18@gmail.com> * Fix build. * Store `DataColumnInfo` in database and various cleanups. * Update `DataColumnSidecar` ssz max size and remove panic code. --------- Co-authored-by: Eitan Seri-Levi <eserilev@gmail.com> Co-authored-by: Jacob Kaufmann <jacobkaufmann18@gmail.com> * feat: add DAS KZG in data col construction (sigp#5210) * feat: add DAS KZG in data col construction * refactor data col sidecar construction * refactor: add data cols to GossipVerifiedBlockContents * Disable windows tests for `das` branch. (c-kzg doesn't build on windows) * Formatting and lint changes only. * refactor: remove iters in construction of data cols * Update vec capacity and error handling. * Add `data_column_sidecar_computation_seconds` metric. --------- Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * Merge branch 'unstable' into das # Conflicts: # .github/workflows/test-suite.yml # beacon_node/lighthouse_network/src/types/topics.rs * fix: update data col subnet count from 64 to 32 (sigp#5413) * feat: add peerdas custody field to ENR (sigp#5409) * feat: add peerdas custody field to ENR * add hash prefix step in subnet computation * refactor test and fix possible u64 overflow * default to min custody value if not present in ENR * Merge branch 'unstable' into das * Merge branch 'unstable' into das-unstable-merge-0415 # Conflicts: # Cargo.lock # beacon_node/beacon_chain/src/data_availability_checker.rs # beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # beacon_node/beacon_chain/src/data_availability_checker/processing_cache.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/network/src/network_beacon_processor/mod.rs # beacon_node/network/src/sync/block_lookups/tests.rs # crypto/kzg/Cargo.toml * Merge remote-tracking branch 'sigp/unstable' into das * Merge remote-tracking branch 'sigp/unstable' into das * Fix merge conflicts. * Send custody data column to `DataAvailabilityChecker` for determining block importability (sigp#5570) * Only import custody data columns after publishing a block. * Add `subscribe-all-data-column-subnets` and pass custody column count to `availability_cache`. * Add custody requirement checks to `availability_cache`. * Fix config not being passed to DAChecker and add more logging. * Introduce `peer_das_epoch` and make blobs and columns mutually exclusive. * Add DA filter for PeerDAS. * Fix data availability check and use test_logger in tests. * Fix subscribe to all data column subnets not working correctly. * Fix tests. * Only publish column sidecars if PeerDAS is activated. Add `PEER_DAS_EPOCH` chain spec serialization. * Remove unused data column index in `OverflowKey`. * Fix column sidecars incorrectly produced when there are no blobs. * Re-instate index to `OverflowKey::DataColumn` and downgrade noisy debug log to `trace`. * DAS sampling on sync (sigp#5616) * Data availability sampling on sync * Address @jimmygchen review * Trigger sampling * Address some review comments and only send `SamplingBlock` sync message after PEER_DAS_EPOCH. --------- Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * Merge branch 'unstable' into das # Conflicts: # Cargo.lock # Cargo.toml # beacon_node/beacon_chain/src/block_verification.rs # beacon_node/http_api/src/publish_blocks.rs # beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs # beacon_node/lighthouse_network/src/rpc/protocol.rs # beacon_node/lighthouse_network/src/types/pubsub.rs # beacon_node/network/src/sync/block_lookups/single_block_lookup.rs # beacon_node/store/src/hot_cold_store.rs # consensus/types/src/beacon_state.rs # consensus/types/src/chain_spec.rs # consensus/types/src/eth_spec.rs * Merge branch 'unstable' into das * Re-process early sampling requests (sigp#5569) * Re-process early sampling requests # Conflicts: # beacon_node/beacon_processor/src/work_reprocessing_queue.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/network/src/network_beacon_processor/rpc_methods.rs * Update beacon_node/beacon_processor/src/work_reprocessing_queue.rs Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * Add missing var * Beta compiler fixes and small typo fixes. * Remove duplicate method. --------- Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * Merge remote-tracking branch 'sigp/unstable' into das * Fix merge conflict. * Add data columns by root to currently supported protocol list (sigp#5678) * Add data columns by root to currently supported protocol list. * Add missing data column by roots handling. * Merge branch 'unstable' into das # Conflicts: # Cargo.lock # Cargo.toml # beacon_node/network/src/sync/block_lookups/tests.rs # beacon_node/network/src/sync/manager.rs * Fix simulator tests on `das` branch (sigp#5731) * Bump genesis delay in sim tests as KZG setup takes longer for DAS. * Fix incorrect YAML spacing. * DataColumnByRange boilerplate (sigp#5353) * add boilerplate * fmt * PeerDAS custody lookup sync (sigp#5684) * Implement custody sync * Lint * Fix tests * Fix rebase issue * Add data column kzg verification and update `c-kzg`. (sigp#5701) * Add data column kzg verification and update `c-kzg`. * Fix incorrect `Cell` size. * Add kzg verification on rpc blocks. * Add kzg verification on rpc data columns. * Rename `PEER_DAS_EPOCH` to `EIP7594_FORK_EPOCH` for client interop. (sigp#5750) * Fetch custody columns in range sync (sigp#5747) * Fetch custody columns in range sync * Clean up todos * Remove `BlobSidecar` construction and publish after PeerDAS activated (sigp#5759) * Avoid building and publishing blob sidecars after PeerDAS. * Ignore gossip blobs with a slot greater than peer das activation epoch. * Only attempt to verify blob count and import blobs before PeerDAS. * sigp#5684 review comments (sigp#5748) * sigp#5684 review comments. * Doc and message update only. * Fix incorrect condition when constructing `RpcBlock` with `DataColumn`s * Make sampling tests deterministic (sigp#5775) * PeerDAS spec tests (sigp#5772) * Add get_custody_columns spec tests. * Add kzg merkle proof spec tests. * Add SSZ spec tests. * Add remaining KZG tests * Load KZG only once per process, exclude electra tests and add missing SSZ tests. * Fix lint and missing changes. * Ignore macOS generated file. * Merge remote branch 'sigp/unstable' into das * Merge remote tracking branch 'origin/unstable' into das * Implement unconditional reconstruction for supernodes (sigp#5781) * Implement unconditional reconstruction for supernodes * Move code into KzgVerifiedCustodyDataColumn * Remove expect * Add test * Thanks justin * Add withhold attack mode for interop (sigp#5788) * Add withhold attack mode * Update readme * Drop added readmes * Undo styling changes * Add column gossip verification and handle unknown parent block (sigp#5783) * Add column gossip verification and handle missing parent for columns. * Review PR * Fix rebase issue * more lint issues :) --------- Co-authored-by: dapplion <35266934+dapplion@users.noreply.github.com> * Trigger sampling on sync events (sigp#5776) * Trigger sampling on sync events * Update beacon_chain.rs * Fix tests * Fix tests * PeerDAS parameter changes for devnet-0 (sigp#5779) * Update PeerDAS parameters to latest values. * Lint fix * Fix lint. * Update hardcoded subnet count to 64 (sigp#5791) * Fix incorrect columns per subnet and config cleanup (sigp#5792) * Tidy up PeerDAS preset and config values. * Fix broken config * Fix DAS branch CI (sigp#5793) * Fix invalid syntax. * Update cli doc. Ignore get_custody_columns test temporarily. * Fix failing test and add verify inclusion test. * Undo accidentally removed code. * Only attempt reconstruct columns once. (sigp#5794) * Re-enable precompute table for peerdas kzg (sigp#5795) * Merge branch 'unstable' into das * Update subscription filter. (sigp#5797) * Remove penalty for duplicate columns (expected due to reconstruction) (sigp#5798) * Revert DAS config for interop testing. Optimise get_custody_columns function. (sigp#5799) * Don't perform reconstruction for proposer node as it already has all the columns. (sigp#5806) * Multithread compute_cells_and_proofs (sigp#5805) * Multi-thread reconstruct data columns * Multi-thread path for block production * Merge branch 'unstable' into das # Conflicts: # .github/workflows/test-suite.yml # beacon_node/network/src/sync/block_lookups/mod.rs # beacon_node/network/src/sync/block_lookups/single_block_lookup.rs # beacon_node/network/src/sync/network_context.rs * Fix CI errors. * Move PeerDAS type-level config to configurable `ChainSpec` (sigp#5828) * Move PeerDAS type level config to `ChainSpec`. * Fix tests * Misc custody lookup improvements (sigp#5821) * Improve custody requests * Type DataColumnsByRootRequestId * Prioritize peers and load balance * Update tests * Address PR review * Merge branch 'unstable' into das * Rename deploy_block in network config (`das` branch) (sigp#5852) * Rename deploy_block.txt to deposit_contract_block.txt * fmt --------- Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com> * Merge branch 'unstable' into das * Fix CI and merge issues. * Merge branch 'unstable' into das # Conflicts: # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # lcli/src/main.rs * Store data columns individually in store and caches (sigp#5890) * Store data columns individually in store and caches * Implement data column pruning * Merge branch 'unstable' into das # Conflicts: # Cargo.lock * Update reconstruction benches to newer criterion version. (sigp#5949) * Merge branch 'unstable' into das # Conflicts: # .github/workflows/test-suite.yml * chore: add `recover_cells_and_compute_proofs` method (sigp#5938) * chore: add recover_cells_and_compute_proofs method * Introduce type alias `CellsAndKzgProofs` to address type complexity. --------- Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * Update `csc` format in ENR and spec tests for devnet-1 (sigp#5966) * Update `csc` format in ENR. * Add spec tests for `recover_cells_and_kzg_proofs`. * Add tests for ENR. * Fix failing tests. * Add protection against invalid csc value in ENR. * Fix lint * Fix csc encoding and decoding (sigp#5997) * Fix data column rpc request not being sent due to incorrect limits set. (sigp#6000) * Fix incorrect inbound request count causing rate limiting. (sigp#6025) * Merge branch 'stable' into das # Conflicts: # beacon_node/network/src/sync/block_lookups/tests.rs # beacon_node/network/src/sync/block_sidecar_coupling.rs # beacon_node/network/src/sync/manager.rs # beacon_node/network/src/sync/network_context.rs # beacon_node/network/src/sync/network_context/requests.rs * Merge remote-tracking branch 'unstable' into das * Add kurtosis config for DAS testing (sigp#5968) * Add kurtosis config for DAS testing. * Fix invalid yaml file * Update network parameter files. * chore: add rust PeerdasKZG crypto library for peerdas functionality and rollback c-kzg dependency to 4844 version (sigp#5941) * chore: add recover_cells_and_compute_proofs method * chore: add rust peerdas crypto library * chore: integrate peerdaskzg rust library into kzg crate * chore(multi): - update `ssz_cell_to_crypto_cell` - update conversion from the crypto cell type to a Vec<u8>. Since the Rust library defines them as references to an array, the conversion is simply `to_vec` * chore(multi): - update rest of code to handle the new crypto `Cell` type - update test case code to no longer use the Box type * chore: cleanup of superfluous conversions * chore: revert c-kzg dependency back to v1 * chore: move dependency into correct order * chore: update rust dependency - This version includes a new method `PeerDasContext::with_num_threads` * chore: remove Default initialization of PeerDasContext and explicitly set the parameters in `new_from_trusted_setup` * chore: cleanup exports * chore: commit updated cargo.lock * Update Cargo.toml Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * chore: rename dependency * chore: update peerdas lib - sets the blst version to 0.3 so that it matches whatever lighthouse is using. Although 0.3.12 is latest, lighthouse is pinned to 0.3.3 * chore: fix clippy lifetime - Rust doesn't allow you to elide the lifetime on type aliases * chore: cargo clippy fix * chore: cargo fmt * chore: update lib to add redundant checks (these will be removed in consensus-specs PR 3819) * chore: update dependency to ignore proofs * chore: update peerdas lib to latest * update lib * chore: remove empty proof parameter --------- Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * Update PeerDAS interop testnet config (sigp#6069) * Update interop testnet config. * Fix typo and remove target peers * Avoid retrying same sampling peer that previously failed. (sigp#6084) * Various fixes to custody range sync (sigp#6004) * Only start requesting batches when there are good peers across all custody columns to avoid spaming block requests. * Add custody peer check before mutating `BatchInfo` to avoid inconsistent state. * Add check to cover a case where batch is not processed while waiting for custody peers to become available. * Fix lint and logic error * Fix `good_peers_on_subnet` always returning false for `DataColumnSubnet`. * Add test for `get_custody_peers_for_column` * Revert epoch parameter refactor. * Fall back to default custody requiremnt if peer ENR is not present. * Add metrics and update code comment. * Add more debug logs. * Use subscribed peers on subnet before MetaDataV3 is implemented. Remove peer_id matching when injecting error because multiple peers are used for range requests. Use randomized custodial peer to avoid repeatedly sending requests to failing peers. Batch by range request where possible. * Remove unused code and update docs. * Add comment * chore: update peerdas-kzg library (sigp#6118) * chore: update peerDAS lib * chore: update library * chore: update library to version that include "init context" benchmarks and optional validation checks * chore: (can remove) -- Add benchmarks for init context * Prevent continuous searchers for low-peer networks (sigp#6162) * Merge branch 'unstable' into das * Fix merge conflicts * Add cli flag to enable sampling and disable by default. (sigp#6209) * chore: Use reference to an array representing a blob instead of an owned KzgBlob (sigp#6179) * add KzgBlobRef type * modify code to use KzgBlobRef * clippy * Remove Deneb blob related changes to maintain compatibility with `c-kzg-4844`. --------- Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * Store computed custody subnets in PeerDB and fix custody lookup test (sigp#6218) * Fix failing custody lookup tests. * Store custody subnets in PeerDB, fix custody lookup test and refactor some methods. * Merge branch 'unstable' into das # Conflicts: # beacon_node/beacon_chain/src/beacon_chain.rs # beacon_node/beacon_chain/src/block_verification_types.rs # beacon_node/beacon_chain/src/builder.rs # beacon_node/beacon_chain/src/data_availability_checker.rs # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # beacon_node/beacon_chain/src/data_column_verification.rs # beacon_node/beacon_chain/src/early_attester_cache.rs # beacon_node/beacon_chain/src/historical_blocks.rs # beacon_node/beacon_chain/tests/store_tests.rs # beacon_node/lighthouse_network/src/discovery/enr.rs # beacon_node/network/src/service.rs # beacon_node/src/cli.rs # beacon_node/store/src/hot_cold_store.rs # beacon_node/store/src/lib.rs # lcli/src/generate_bootnode_enr.rs * Fix CI failures after merge. * Batch sampling requests by peer (sigp#6256) * Batch sampling requests by peer * Fix clippy errors * Fix tests * Add column_index to error message for ease of tracing * Remove outdated comment * Fix range sync never evaluating request as finished, causing it to get stuck. (sigp#6276) * Merge branch 'unstable' into das-0821-merge # Conflicts: # Cargo.lock # Cargo.toml # beacon_node/beacon_chain/src/beacon_chain.rs # beacon_node/beacon_chain/src/data_availability_checker.rs # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # beacon_node/beacon_chain/src/data_column_verification.rs # beacon_node/beacon_chain/src/kzg_utils.rs # beacon_node/beacon_chain/src/metrics.rs # beacon_node/beacon_processor/src/lib.rs # beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs # beacon_node/lighthouse_network/src/rpc/config.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/lighthouse_network/src/rpc/outbound.rs # beacon_node/lighthouse_network/src/rpc/rate_limiter.rs # beacon_node/lighthouse_network/src/service/api_types.rs # beacon_node/lighthouse_network/src/types/globals.rs # beacon_node/network/src/network_beacon_processor/mod.rs # beacon_node/network/src/network_beacon_processor/rpc_methods.rs # beacon_node/network/src/network_beacon_processor/sync_methods.rs # beacon_node/network/src/sync/block_lookups/common.rs # beacon_node/network/src/sync/block_lookups/mod.rs # beacon_node/network/src/sync/block_lookups/single_block_lookup.rs # beacon_node/network/src/sync/block_lookups/tests.rs # beacon_node/network/src/sync/manager.rs # beacon_node/network/src/sync/network_context.rs # consensus/types/src/data_column_sidecar.rs # crypto/kzg/Cargo.toml # crypto/kzg/benches/benchmark.rs # crypto/kzg/src/lib.rs * Fix custody tests and load PeerDAS KZG instead. * Fix ef tests and bench compilation. * Fix failing sampling test. * Merge pull request sigp#6287 from jimmygchen/das-0821-merge Merge `unstable` into `das` 20240821 * Remove get_block_import_status * Merge branch 'unstable' into das * Re-enable Windows release tests. * Address some review comments. * Address more review comments and cleanups. * Comment out peer DAS KZG EF tests for now * Address more review comments and fix build. * Merge branch 'das' of github.com:sigp/lighthouse into das * Unignore Electra tests * Fix metric name * Address some of Pawan's review comments * Merge remote-tracking branch 'origin/unstable' into das * Update PeerDAS network parameters for peerdas-devnet-2 (sigp#6290) * update subnet count & custody req * das network params * update ef tests --------- Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

jimmygchen added work-in-progress PR is a work-in-progress das Data Availability Sampling das-devnet-1 labels Jun 27, 2024

Only start requesting batches when there are good peers across all cu…

edc584a

…stody columns to avoid spaming block requests.

jimmygchen force-pushed the custody-sync-peers branch from 85a51f6 to edc584a Compare June 27, 2024 00:40

Add custody peer check before mutating BatchInfo to avoid inconsist…

9f82497

…ent state.

jimmygchen force-pushed the custody-sync-peers branch from 847c4f2 to 9f82497 Compare June 27, 2024 02:58

Add check to cover a case where batch is not processed while waiting …

05608b0

…for custody peers to become available.

jimmygchen force-pushed the custody-sync-peers branch from 0d97026 to 05608b0 Compare June 27, 2024 04:15

Fix lint and logic error

9f20029

jimmygchen added ready-for-review The code is ready for review and removed work-in-progress PR is a work-in-progress labels Jun 27, 2024

jimmygchen added 3 commits June 27, 2024 15:52

Fix good_peers_on_subnet always returning false for `DataColumnSubn…

4079d2e

…et`.

Merge branch 'das' into custody-sync-peers

de05355

Add test for get_custody_peers_for_column

876ea3b

jimmygchen added work-in-progress PR is a work-in-progress and removed ready-for-review The code is ready for review labels Jun 27, 2024

dapplion reviewed Jun 28, 2024

View reviewed changes

beacon_node/network/src/sync/range_sync/chain.rs Outdated Show resolved Hide resolved

beacon_node/network/src/sync/range_sync/chain.rs Show resolved Hide resolved

Revert epoch parameter refactor.

4373a28

jimmygchen changed the title ~~Avoid spamming blocks by range request until there's enough peers on all custody subnets~~ Avoid spamming blocks by range request until there's available peer on all custody subnets Jul 1, 2024

Fall back to default custody requiremnt if peer ENR is not present.

03a9dce

jimmygchen force-pushed the custody-sync-peers branch from 014c994 to 03a9dce Compare July 1, 2024 02:39

jimmygchen added ready-for-review The code is ready for review and removed work-in-progress PR is a work-in-progress labels Jul 1, 2024

jimmygchen force-pushed the custody-sync-peers branch from ede964e to 129ac92 Compare July 1, 2024 02:54

jimmygchen mentioned this pull request Jul 1, 2024

peerdas-devnet-1 testing branch #6023

Closed

Add metrics and update code comment.

cfb3ebc

jimmygchen force-pushed the custody-sync-peers branch from 129ac92 to cfb3ebc Compare July 1, 2024 06:29

jimmygchen added 3 commits July 2, 2024 11:53

Add more debug logs.

0a27146

Merge branch 'das' into custody-sync-peers

9cfbfa6

jimmygchen changed the title ~~Avoid spamming blocks by range request until there's available peer on all custody subnets~~ Various fixes to custody range sync Jul 2, 2024

jimmygchen force-pushed the custody-sync-peers branch from abf8b2a to bf70fb4 Compare July 2, 2024 07:52

Remove unused code and update docs.

c10fe36

jimmygchen force-pushed the custody-sync-peers branch from ac981a0 to c10fe36 Compare July 2, 2024 08:05

jimmygchen commented Jul 8, 2024

View reviewed changes

jimmygchen added work-in-progress PR is a work-in-progress and removed ready-for-review The code is ready for review labels Jul 10, 2024

jimmygchen mentioned this pull request Jul 12, 2024

Testing branch for peerdas-devnet-2 #6081

Closed

jimmygchen added 2 commits July 15, 2024 10:55

Merge branch 'das' into custody-sync-peers

9cfc670

# Conflicts: # beacon_node/network/src/sync/network_context.rs # beacon_node/network/src/sync/range_sync/batch.rs

Add comment

6ff408f

jimmygchen added a commit that referenced this pull request Jul 15, 2024

Various fixes to custody range sync (#6004)

84fdf0f

# Conflicts: # beacon_node/network/src/sync/sampling.rs

dapplion approved these changes Jul 16, 2024

View reviewed changes

Merge branch 'das' into custody-sync-peers

e04f98d

# Conflicts: # beacon_node/network/src/sync/sampling.rs

jimmygchen merged commit 55a3be7 into sigp:das Jul 16, 2024
6 of 7 checks passed

jimmygchen mentioned this pull request Jul 16, 2024

Investigate stuck / slow range sync on PeerDAS networks #6113

Closed

This was referenced Aug 15, 2024

Improve range sync with PeerDAS #6258

Open

Fix PeerDAS stuck range sync when multiple columns are batched in a single request #6276

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Various fixes to custody range sync #6004

Various fixes to custody range sync #6004

jimmygchen commented Jun 27, 2024 •

edited

Loading

jimmygchen commented Jun 27, 2024

dapplion left a comment

jimmygchen Jul 8, 2024

jimmygchen Jul 11, 2024

jimmygchen Jul 12, 2024

jimmygchen Jul 15, 2024

jimmygchen Jul 15, 2024

Various fixes to custody range sync #6004

Various fixes to custody range sync #6004

Conversation

jimmygchen commented Jun 27, 2024 • edited Loading

Issue Addressed

jimmygchen commented Jun 27, 2024

dapplion left a comment

Choose a reason for hiding this comment

jimmygchen Jul 8, 2024

Choose a reason for hiding this comment

jimmygchen Jul 11, 2024

Choose a reason for hiding this comment

jimmygchen Jul 12, 2024

Choose a reason for hiding this comment

jimmygchen Jul 15, 2024

Choose a reason for hiding this comment

jimmygchen Jul 15, 2024

Choose a reason for hiding this comment

jimmygchen commented Jun 27, 2024 •

edited

Loading