Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(bin): warn about CL health only when pipeline is idle #4835

Merged
merged 3 commits into from
Sep 28, 2023

Conversation

shekhirin
Copy link
Collaborator

No description provided.

@shekhirin shekhirin added C-enhancement New feature or request A-observability Related to tracing, metrics, logs and other observability tools labels Sep 28, 2023
@shekhirin shekhirin requested a review from onbjerg as a code owner September 28, 2023 11:05
@shekhirin shekhirin requested a review from mattsse September 28, 2023 11:05
@codecov
Copy link

codecov bot commented Sep 28, 2023

Codecov Report

Merging #4835 (6f67025) into main (8e9937b) will decrease coverage by 0.03%.
Report is 1 commits behind head on main.
The diff coverage is 0.00%.

Impacted file tree graph

Files Coverage Δ
bin/reth/src/node/events.rs 11.21% <0.00%> (-0.21%) ⬇️

... and 7 files with indirect coverage changes

Flag Coverage Δ
integration-tests 16.12% <0.00%> (-0.02%) ⬇️
unit-tests 63.29% <0.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
reth binary 32.06% <0.00%> (-0.03%) ⬇️
blockchain tree 83.69% <ø> (ø)
pipeline 88.45% <ø> (ø)
storage (db) 73.23% <ø> (ø)
trie 94.48% <ø> (-0.04%) ⬇️
txpool 49.99% <ø> (-0.02%) ⬇️
networking 76.44% <ø> (-0.13%) ⬇️
rpc 57.78% <ø> (-0.01%) ⬇️
consensus 62.59% <ø> (ø)
revm 28.48% <ø> (ø)
payload builder 8.16% <ø> (ø)
primitives 86.22% <ø> (ø)

Copy link
Collaborator

@mattsse mattsse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need clarification on comment

}
ConsensusLayerHealthEvent::HaveNotReceivedUpdatesForAWhile(period) => {
warn!(?period, "Beacon client online, but no consensus updates received for a while. Please fix your beacon client to follow the chain!")
// If pipeline is running, consensus layer messages are expected to be neither received nor
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not correct, they're always handled.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, removed the part about processing. We always process incoming messages, but not expect to receive them when the pipeline is running.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also don't think this is correct, we're still getting FCUs, at least with lighthouse, so perhaps this is a client specific thing.

the change makes sense, but comment should reflect this

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, maybe "If pipeline is running, it's fine to not receive any consensus layer messages."?

}
ConsensusLayerHealthEvent::HaveNotReceivedUpdatesForAWhile(period) => {
warn!(?period, "Beacon client online, but no consensus updates received for a while. Please fix your beacon client to follow the chain!")
// If pipeline is running, no consensus layer messages are expected to be received.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry to bikeshed here, but I think this is still not accurate because we know that some clients still send us messages.

this comment should reflect that it can be the case not that it is expected

@mattsse mattsse enabled auto-merge September 28, 2023 14:10
@mattsse mattsse added this pull request to the merge queue Sep 28, 2023
Merged via the queue into main with commit a4e7760 Sep 28, 2023
@mattsse mattsse deleted the alexey/cl-health-pipeline-idle branch September 28, 2023 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-observability Related to tracing, metrics, logs and other observability tools C-enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants