Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Horizon metric for keeping track of slow ingestion restarts #5417

Closed
tamirms opened this issue Aug 8, 2024 · 0 comments · Fixed by #5449
Closed

Add Horizon metric for keeping track of slow ingestion restarts #5417

tamirms opened this issue Aug 8, 2024 · 0 comments · Fixed by #5449

Comments

@tamirms
Copy link
Contributor

tamirms commented Aug 8, 2024

While rolling out the limited history horizon instance to production we realized that some of the production ingesting horizon nodes were configured to run captive core with BucketDB disabled. Consequently, captive core was running with in-memory mode and this resulted in Horizon being very slow to resume ingestion after restarts.

We should have a metric which is incremented whenever captive-core has to catch up from scratch instead of quickly resuming from the LCL recorded in BucketDB. Once this metric is in place, we can modify our release testing checklist to make sure that the release branch of horizon does not regress by restarting captive-core from scratch at an unusually high rate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

1 participant