Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unnecessary repeated epoch transitions when node is stuck #3099

Closed
dapplion opened this issue Sep 9, 2021 · 1 comment
Closed

Unnecessary repeated epoch transitions when node is stuck #3099

dapplion opened this issue Sep 9, 2021 · 1 comment
Labels
prio-medium Resolve this some time soon (tm). scope-performance Performance issue and ideas to improve performance.

Comments

@dapplion
Copy link
Contributor

dapplion commented Sep 9, 2021

Describe the bug

Due to the invalid head state root issue, our Prater node got stuck for a few days in limbo, memory is stable. However, many epoch transitions are done per epoch (32-50). Maybe the regen is not caching the state after advancing it? Or doesn't know how to query it.

Check in the REST API metrics the only external interaction with the node is getProposerDuties, there was no gossip received during that period.

Screenshot from 2021-09-09 19-39-55

Screenshot from 2021-09-09 19-40-15

Screenshot from 2021-09-09 19-40-26

Note: prometheus query for epoch transitions per epoch

increase(lodestar_stfn_epoch_transition_seconds_count[384s])

Expected behavior

Steps to Reproduce

Screenshots

Desktop (please complete the following information):

  • OS:
  • Version:
  • Branch:
  • Commit hash:
@q9f q9f added mod1-beaconchain prio-medium Resolve this some time soon (tm). scope-performance Performance issue and ideas to improve performance. and removed mod1-beaconchain labels Sep 14, 2021
@dapplion
Copy link
Contributor Author

Closing issues for specific incidents on old versions of Lodestar, please re-open if it happens with latest stable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
prio-medium Resolve this some time soon (tm). scope-performance Performance issue and ideas to improve performance.
Projects
None yet
Development

No branches or pull requests

2 participants