Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Single lookup improvements #5488

Merged
merged 15 commits into from
Mar 27, 2024
Merged

Single lookup improvements #5488

merged 15 commits into from
Mar 27, 2024

Conversation

realbigsean
Copy link
Member

Includes #5484 and #5477 (modulo the PR feedback)

Copied from #5484:

Issue Addressed

  • add logs
  • tighten up lookup de-duplication logic
  • an alternative fix to: Fix unexpected UnrequestedBlobId and ExtraBlocksReturned errors due to race conditions. #5477
    • the core issue in the bug there is that we update requested_ids while blobs are still downloading if we download the RPC block in between blobs of the stream. this becomes and issue because requested_ids is updated with any blobs that come via gossip.
    • proposed solution here is to only update the blobs if the request state is AwaitingDownload which means it has been either completed or not yet started

@realbigsean realbigsean added the ready-for-review The code is ready for review label Mar 26, 2024
@realbigsean realbigsean mentioned this pull request Mar 26, 2024
Closed
@realbigsean
Copy link
Member Author

closing in favor of #5488

@realbigsean realbigsean reopened this Mar 26, 2024
@realbigsean
Copy link
Member Author

shoot I closed the wrong one

@realbigsean realbigsean mentioned this pull request Mar 26, 2024
Closed
@realbigsean realbigsean added bug Something isn't working deneb v5.2.0 Q2 2024 labels Mar 26, 2024
jimmygchen
jimmygchen approved these changes Mar 27, 2024
View reviewed changes
Copy link
Member

@jimmygchen jimmygchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! 👍

@jimmygchen jimmygchen added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Mar 27, 2024
@realbigsean
Copy link
Member Author

@Mergifyio queue

@mergify Mergify
Copy link

mergify bot commented Mar 27, 2024 •
edited
Loading

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 334aa2e

@mergify mergify bot merged commit 334aa2e into sigp:unstable Mar 27, 2024
29 checks passed
@realbigsean realbigsean mentioned this pull request Mar 27, 2024
Merged
jimmygchen added a commit that referenced this pull request Apr 9, 2024
@jimmygchen @chong-he
@realbigsean @eserilev @joaolago1113 @ackintosh @jxs @michaelsproul @threewebcode @zhiqiangxu @dapplion @danielrachi1 @dknopik @pawanjay176 @macladson @0xalex88 @GeemoCandama
Merge latest unstable to das branch (#5538)
5cb1b8e 
* Add attestation simulator, blobs info and some updates to Lighthouse Book (#5364)

* Apply suggestions from code review

* Revise attestation simulator doc

* Revise blobs.md

* Summary

* Add blobs

* Simulator docs

* Revise attestation simulator

* minor formatting

* Revise vm node

* Update faq

* Update faq

* Add link to v4.6.0

* Remove minification in the docs

* Update Goerli to Holesky

* Add a note on moved vm validator monitor

* Update Rpi 4 note

* Revise attestation simulator doc

* Add docs for attestation simulator

* update database table

* Update faq on resources used

* Fix and update table

* Dedup parent blob requests (#5432)

* de dup parent blob requests

* add new line

* Fix Rust beta compiler errors 1.78.0-beta.1 (#5439)

* remove redundant imports

* fix test

* contains key

* fmt

* Merge branch 'unstable' into fix-beta-compiler

* Improvements and Fixes in Documentation, Including Corrected Command Usage (#4998)

* Fix typo: change 'periodical' to 'periodic' in progress updates description

* Fix wrong command in Usage section

* fix typo in Development Environment section

* Fix typo: change 'Explictly' to 'Explicitly'

* Fix typos in Lighthouse UI and Contributing sections

* Fix typo: replace 'confirms' with 'conforms' in Beacon Node API description

* fix minor typographical error: change 'advice' to 'advise' in SIGILL warning message

* Fix spelling error in Detailed Guide section

* Revert "Fix typo: change 'Explictly' to 'Explicitly'"

This reverts commit 6b07816.

* Revert "fix minor typographical error: change 'advice' to 'advise' in SIGILL warning message"

This reverts commit a4904a0.

* compiled

* Revert "compiled"

This reverts commit 425a553.

* Revert "Revert "compiled""

This reverts commit b1f871c.

* Empty commit to trigger CI.

* Fix peer count metrics (#5404)

* Set the peers_per_client metrics directly, rather than using increment/decrement

* Move PEERS_CONNECTED related update to the same place

* Move PEERS_CONNECTED_MULTI related update to the same place

* Rename

* Remove unused variables

* fix NAT `nat_open` metrics report (#5427)

* fix nat reporting

* Fix gossip verification of duplicate attester slashings (#5385)

* Fix gossip verification of duplicate attester slashings

* disable libp2p upnp (#5449)

* disable libp2p upnp

when passing --disable-upnp cli flag

* Release v5.1.2 (#5453)

* Release v5.1.2

* Improve parent lookup logging (#5451)

* upgrade parent lookup result processing logs to debug, use display instead of debug for BlockError in case a blob parent unknown error is hit, add block root to BlockIsAlreadyKnown

* fix compile

* fix compile

* fix compile

* fix: tail command typo (#5456)

* fix: tail command typo

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into patch-1

* chore: remove stale comment (#5440)

* rm stale comment

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into rm_irelevant_comment

* chore: reduce scope of commitment (#5426)

* reduce scope of commitment

* avoid clone for last reference

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into mod_merge_single_blob

* Fix one and hide all beacon-processor flags (#5397)

* Fix `beacon-processor-work-queue-len`

* Hide beacon-processor flags

* Delete unused incomplete_processing_components (#5418)

* Delete unused incomplete_processing_components

* lint

* Make sure all geth processes are killed when stopping a local testnet (#5383)

* Fix geth processes not being killed when stopping a local testnet

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into fix_stop_testnet

* Verify whether validators really are unknown during sync committee duty API request (#5174)

* Verify whether validators really are unknown during sync committee duty API request

* Merge branch 'unstable' into fix-4717

* Merge branch 'unstable' into fix-4717

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into fix-4717

* Built-in documentation text width in Lighthouse book (#5394)

* Adjust width

* Commit changes

* Trigger Build

* Delete PRE_CAPELLA_ENGINE_CAPABILITIES (#5406)

* Adjust width

* Commit changes

* Delete PRE_CAPELLA_ENGINE_CAPABILITIES

* Revert "Adjust width"

This reverts commit 6fea81b.

* Revert "Commit changes"

This reverts commit d00859a.

* Simplify

* Merge branch 'delete-pre-capella' of https://github.com/chong-he/lighthouse into delete-pre-capella

* Add Capella & Deneb light client support  (#4946)

* rebase and add comment

* conditional test

* test

* optimistic chould be working now

* finality should be working now

* try again

* try again

* clippy fix

* add lc bootstrap beacon api

* add lc optimistic/finality update to events

* fmt

* That error isn't occuring on my computer but I think this should fix it

* Merge branch 'unstable' into light_client_beacon_api_1

# Conflicts:
#	beacon_node/beacon_chain/src/events.rs
#	beacon_node/http_api/src/lib.rs
#	beacon_node/http_api/src/test_utils.rs
#	beacon_node/http_api/tests/main.rs
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/methods.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	beacon_node/network/src/beacon_processor/worker/rpc_methods.rs
#	beacon_node/tests/test.rs
#	common/eth2/src/types.rs
#	lighthouse/src/main.rs

* Add missing test file

* Update light client types to comply with Altair light client spec.

* Fix test compilation

* Merge branch 'unstable' into light_client_beacon_api_1

* Support deserializing light client structures for the Bellatrix fork

* Move `get_light_client_bootstrap` logic to `BeaconChain`. `LightClientBootstrap` API to return `ForkVersionedResponse`.

* Misc fixes.
- log cleanup
- move http_api config mutation to `config::get_config` for consistency
- fix light client API responses

* Add light client bootstrap API test and fix existing ones.

* Merge branch 'unstable' into light_client_beacon_api_1

* Fix test for `light-client-server` http api config.

* Appease clippy

* Add Altair light client SSZ tests

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into light_client_beacon_api_1

* updates to light client header

* light client header from signed beacon block

* using options

* implement helper functions

* placeholder conversion from vec hash256 to exec branch

* add deneb

* using fixed vector

* remove unwraps

* by epoch

* compute merkle proof

* merkle proof

* update comments

* resolve merge conflicts

* linting

* Merge branch 'unstable' into light-client-ssz-tests

# Conflicts:
#	beacon_node/beacon_chain/src/beacon_chain.rs
#	consensus/types/src/light_client_bootstrap.rs
#	consensus/types/src/light_client_header.rs

* superstruct attempt

* superstruct changes

* lint

* altair

* update

* update

* changes to light_client_optimistic_ and finality

* merge unstable

* refactor

* resolved merge conflicts

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types

* block_to_light_client_header fork aware

* fmt

* comment fix

* comment fix

* include merge fork, update deserialize_by_fork, refactor

* fmt

* pass by ref to prevent clone

* rename merkle proof fn

* add FIXME

* LightClientHeader TestRandom

* fix comments

* fork version deserialize

* merge unstable

* move fn arguments, fork name calc

* use task executor

* remove unneeded fns

* remove dead code

* add manual ssz decoding/encoding and add ssz_tests_by_fork macro

* merge deneb types with tests

* merge ssz tests, revert code deletion, cleanup

* move chainspec

* update ssz tests

* fmt

* light client ssz tests

* change to superstruct

* changes from feedback

* linting

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types

* test fix

* cleanup

* Remove unused `derive`.

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into capella_deneb_light_client_types

* beta compiler fix

* merge

* Fix double counted metrics (#5476)

* Fix double counted metrics

* Move gossipsub into a separate crate (#5401)

* move gossipsub into a separate crate

* Merge branch 'unstable' of github.com:sigp/lighthouse into separate-gossipsub

* address review 2

* clippy beta

* update logging to log gossipsub logs

* Run fork choice after RPC blob import (#5475)

* Run fork choice after RPC blob import

* Single lookup improvements (#5488)

* Fix unexpected `UnrequestedBlobId` and `ExtraBlocksReturned` errors due to race conditions.

* Continue chain segment processing and skip any blocks that are already known, rather than returning an error.

* more de-dup checking

* ensure we don't reset `requested_ids` during rpc download

* better fix

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into more-dup-lookup-fixes

* remove chain hash check

* Merge branch 'fix-block-lookup-race' of https://github.com/jimmygchen/lighthouse into sean-test-lookups

* remove block check

* add back tests

* Log and CI fixes

* undue extra check

* Merge branch 'sean-test-lookups' of https://github.com/realbigsean/lighthouse into sean-test-lookups

* log improvements

* Improve logging

* Lookup log improvements (#5491)

* log improvements

* Bump duplicate cache time (#5493)

* Bump seen_ttl for gossipsub duplicate cache

* Release v5.1.3 (#5497)

* Release v5.1.3

* Add Electra fork boilerplate (#5122)

* Add Electra fork boilerplate

* Remove electra from spec tests

* Fix tests

* Remove sneaky log file

* Fix more tests

* Fix even more tests and add suggestions

* Remove unrelated lcli addition

* Update more tests

* Merge branch 'unstable' into electra

* Add comment for test-suite lcli override

* Merge branch 'unstable' into electra

* Cleanup

* Merge branch 'unstable' into electra

* Apply suggestions

* Merge branch 'unstable' into electra

* Merge sigp/unstable into electra

* Merge branch 'unstable' into electra

* Use `E` for  `EthSpec` globally (#5264)

* Use `E` for `EthSpec` globally

* Fix tests

* Merge branch 'unstable' into e-ethspec

* Merge branch 'unstable' into e-ethspec

# Conflicts:
#	beacon_node/execution_layer/src/engine_api.rs
#	beacon_node/execution_layer/src/engine_api/http.rs
#	beacon_node/execution_layer/src/engine_api/json_structures.rs
#	beacon_node/execution_layer/src/test_utils/handle_rpc.rs
#	beacon_node/store/src/partial_beacon_state.rs
#	consensus/types/src/beacon_block.rs
#	consensus/types/src/beacon_block_body.rs
#	consensus/types/src/beacon_state.rs
#	consensus/types/src/config_and_preset.rs
#	consensus/types/src/execution_payload.rs
#	consensus/types/src/execution_payload_header.rs
#	consensus/types/src/light_client_optimistic_update.rs
#	consensus/types/src/payload.rs
#	lcli/src/parse_ssz.rs

* Bump `h2` for RUSTSEC-2024-0332 (#5514)

* Bump `h2` for RUSTSEC-2024-0332

* Return `not synced` errors for endpoints that require syncing (#5136)

* add not synced filter into then blocks

* refactor

* Single-pass epoch processing and optimised block processing (#5279)

* Single-pass epoch processing (#4483, #4573)

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Delete unused epoch processing code (#5170)

* Delete unused epoch processing code

* Compare total deltas

* Remove unnecessary apply_pending

* cargo fmt

* Remove newline

* Use epoch cache in block packing (#5223)

* Remove progressive balances mode (#5224)

* inline inactivity_penalty_quotient_for_state

* drop previous_epoch_total_active_balance

* fc lint

* spec compliant process_sync_aggregate (#15)

* spec compliant process_sync_aggregate

* Update consensus/state_processing/src/per_block_processing/altair/sync_committee.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Delete the participation cache (#16)

* update help

* Fix op_pool tests

* Fix fork choice tests

* Merge remote-tracking branch 'sigp/unstable' into epoch-single-pass

* Simplify exit cache (#5280)

* Fix clippy on exit cache

* Clean up single-pass a bit (#5282)

* Address Mark's review of single-pass (#5386)

* Merge remote-tracking branch 'origin/unstable' into epoch-single-pass

* Address Sean's review comments (#5414)

* Address most of Sean's review comments

* Simplify total balance cache building

* Clean up unused junk

* Merge remote-tracking branch 'origin/unstable' into epoch-single-pass

* More self-review

* Merge remote-tracking branch 'origin/unstable' into epoch-single-pass

* Merge branch 'unstable' into epoch-single-pass

* Fix imports for beta compiler

* Fix tests, probably

* Remove DataAvailabilityView trait from ChildComponents (#5421)

* Remove DataAvailabilityView trait from ChildComponents

* PR reviews

* Update beacon_node/network/src/sync/block_lookups/common.rs

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into child_components_independent

* Add `is_parent_strong` proposer re-org check (#5417)

* initial fork choice additions

* add helper fns

* add is_parent_strong

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into add_is_parent_strong_check

* disabling proposer reorg should set parent_threshold to u64 max

* add new flag, is_parent_strong check in override fcu params

* cherry-pick changes

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into add_is_parent_strong_check

* cleanup

* fmt

* Minor review tweaks

* Add missing header to `eth/v1/builder/blinded_blocks` (#5407)

* add missing header

* read header in mock builder

* Merge branch 'unstable' into builder-blinded-blocks-missing-header

* Delete `ParticipationCache` (#5525)

* Delete `ParticipationCache`

* Use hashset to filter validators ids in http_api (#5468)

* Use hashset to filter validators ids in http_api

* Update beacon_node/http_api/src/validators.rs

* Support `LightClientFinalityUpdate` and `LightClientOptimisticUpdate` rpcs (#3849)

* add light client optimistic and finality update rpc

* Arc the updates in the response

* add conditional advertisement for both LightClientOptimisticUpdate and LightClientFinalityUpdate

* alter display for inboundrequest light client optimistic and finality updates

* remove LightClientOptimistic/FinalityReuest struct and some minor fixes

* rebase

* failing rpc_test for LightClientBootstrap and beginning of MockLib2pLightClient

* minor change

* added MockRPCHandler by importing everything except OutboundRequest. Need to implement the ConnectionHandler trait now should be copy pastable

* almost there but ran into issue where needed to implement BaseOutboundRequest.

* failing but running with a light client service of sorts

* small test change

* changed Protocol::LightClientBootstrap response limit

* deleted some stuff from ConnectionHandler Implementation for the mock light client if you need to make something with multiple requests work maybe check here

* deleted purging expired inbound/outbound streams code

* deleted drive inbound streams that need to be processed

* removed unused imports

* made things private again

* deleted inject_fully_negotiated_inbound

* made more things private again

* more

* turned the logger off in the test

* added failing test for new rpc

* add rate limit for new rpcs

* change InboundUpgrade function to use new rpcs. fmt. add test for LightClientFinalityUpdate

* rebasing fix

* add LightClientUpdate to handle_rpc functions

* added context bytes

* fmt

* use correct unsed_tcp4_port function

* fix for recent config changes and adding context_bytes for the light client protocols

* fix clippy complaint

* Merge branch 'unstable' into lc-reqresp

# Conflicts:
#	beacon_node/beacon_processor/src/lib.rs
#	beacon_node/lighthouse_network/src/peer_manager/mod.rs
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/config.rs
#	beacon_node/lighthouse_network/src/rpc/methods.rs
#	beacon_node/lighthouse_network/src/rpc/mod.rs
#	beacon_node/lighthouse_network/src/rpc/outbound.rs
#	beacon_node/lighthouse_network/src/rpc/protocol.rs
#	beacon_node/lighthouse_network/src/rpc/rate_limiter.rs
#	beacon_node/lighthouse_network/src/rpc/self_limiter.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	beacon_node/lighthouse_network/tests/common/mod.rs
#	beacon_node/lighthouse_network/tests/rpc_tests.rs
#	beacon_node/network/src/network_beacon_processor/rpc_methods.rs
#	beacon_node/network/src/router.rs

* Error handling updates and various cleanups.

* Moar minor clean ups.

* Do not ban peer for rate limiting light client requests

* Merge branch 'unstable' into lc-reqresp. Also removed the mock light client tests to make it compile (See #4940).

# Conflicts:
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/methods.rs
#	beacon_node/lighthouse_network/src/rpc/mod.rs
#	beacon_node/lighthouse_network/src/rpc/protocol.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	beacon_node/lighthouse_network/tests/common/mod.rs
#	beacon_node/network/src/network_beacon_processor/rpc_methods.rs
#	beacon_node/network/src/router.rs
#	consensus/types/src/light_client_bootstrap.rs
#	consensus/types/src/light_client_finality_update.rs
#	consensus/types/src/light_client_optimistic_update.rs

* Remove unnecessary changes

* Add missing light client queue handling.

* Merge branch 'unstable' into lc-reqresp

* Merge branch 'unstable' into lc-reqresp

# Conflicts:
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	consensus/types/src/light_client_finality_update.rs
#	consensus/types/src/light_client_optimistic_update.rs

* Add context bytes for light client RPC responses.

* Add RPC limits for light client object.

* Fix lint

* Fix incorrect light client max size computation.

* Merge branch 'unstable' into lc-reqresp

# Conflicts:
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/protocol.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs

* Remove unwanted local changes.

* Merge branch 'unstable' into lc-reqresp

* Replace `unimplemented` electra code path with deneb values.

---------

Co-authored-by: chonghe <44791194+chong-he@users.noreply.github.com>
Co-authored-by: realbigsean <sean@sigmaprime.io>
Co-authored-by: Eitan Seri-Levi <eserilev@ucsc.edu>
Co-authored-by: joao <22820692+joaolago1113@users.noreply.github.com>
Co-authored-by: Akihito Nakano <sora.akatsuki@gmail.com>
Co-authored-by: João Oliveira <hello@jxs.pt>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Afanti <127061691+threewebcode@users.noreply.github.com>
Co-authored-by: zhiqiangxu <652732310@qq.com>
Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>
Co-authored-by: Daniel Ramírez-Chiquillo <hi@danielrachi.com>
Co-authored-by: dknopik <107140945+dknopik@users.noreply.github.com>
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-authored-by: Mac L <mjladson@pm.me>
Co-authored-by: 0xalex88 <113263502+0xalex88@users.noreply.github.com>
Co-authored-by: GeemoCandama <104614073+GeemoCandama@users.noreply.github.com>
Hanae1963

This comment was marked as spam.

Sign in to view

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Hanae1963 Hanae1963 Hanae1963 left review comments

@jimmygchen jimmygchen jimmygchen approved these changes

Assignees
No one assigned
Labels
bug Something isn't working deneb ready-for-merge This PR is ready to merge. v5.2.0 Q2 2024
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@realbigsean @jimmygchen @Hanae1963