Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multi: use bitcoind uptime api for healthcheck #4755

Merged
merged 1 commit into from
Nov 12, 2020

Conversation

carlaKC
Copy link
Collaborator

@carlaKC carlaKC commented Nov 9, 2020

Previously we were using GetBestBlock for all of our healthchecks. This was running into some issues with bitcoind, because the GetBestBlock rpc call uses the csmain lock, which is widely used, and would block for extended periods of time (5 minutes sometimes!). btcd has more granular locking, so it does not face these issues.

This PR updates our chain backend health check to use the uptime api because it has no locking. Another option would be to use something like estimatesmartfee which uses a less common lock in bitcoind and does not need us to make an ad-hoc rpcclient. Decided against this because estimatesmartfee could crunch through mempool data unnecessarily.

Fixes #4669.

return nil, err
}
cc.HealthCheck = func() error {
_, err := chainConn.RawRequest("uptime", nil)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which version of bitcoind was this call added with? Perhaps we should do a version check on start up to warn the user about the feature, possibly falling back to the older way of doing things?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First added in 0.15, will add a version check 👍

Copy link
Collaborator Author

@carlaKC carlaKC Nov 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed offline. Actual segwit code in in 0.13, but without activation, 0.14 has no mention of segwit and "full support" added in 0.16, so I think it's safe to assume people have >=0.16 running because you couldn't signal segwit support without it? Can add a version check to be safe if we're worried about it.

@carlaKC carlaKC requested review from Roasbeef and guggero and removed request for cfromknecht November 10, 2020 14:49
The getblockchaininfo call in bitcoind uses a commonly used lock,
csmain, in bitcoind. This made the endpoint unsuitable for a health
check, because some nodes were seeing waits up to 5 minutes (!). This
commit updates our health check function to use the uptime api, provided
our bitcoind version is > 0.15, when the api was added. We do not need
to switch our health check for btcd, because it has more granular
locking.
@carlaKC
Copy link
Collaborator Author

carlaKC commented Nov 11, 2020

Updated this to have a version check to be safe, I don't think it costs us much. It's difficult to tell where the line is for segwit enabled because 0.13 has all the code, but then activation was more complex. All the 0.14 versions are also vulnerable to inflation bug, so I think most people should be running on 0.15 and above.

@Roasbeef Roasbeef added this to the 0.12.0 milestone Nov 12, 2020
Copy link
Member

@Roasbeef Roasbeef left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🍬

@Roasbeef Roasbeef merged commit 05592ad into lightningnetwork:master Nov 12, 2020
@C-Otto
Copy link
Contributor

C-Otto commented Nov 12, 2020

Thanks, that was quick 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Health check: chain backend failed after 3 calls. How to avoid the shutdown?
4 participants