Trying to insert block with invalid number #2136
Labels
I2-bug
The node fails to follow expected behavior.
I10-unconfirmed
Issue might be valid, but it's not yet known.
Comments
nazar-pc
added
I10-unconfirmed
|
This was happening in bursts during what appeared to be a period of increased forking rate. Same node is still online, works fine and doesn't seem to print those messages anymore. |
|
Could it be that there's more than 32 blocks with the same number in the network? Thre's a hard limit for that. There should be a "Too many sibling blocks" error in the log though. Do you have logs for the time period where these errors first started appearing? |
|
There was no such logs, here is a longer snippet: |
|
I've gotten similar, but with some notable differences. Whereas in your log it errors on trying different invalid block numbers, mine got stuck repeatedly erroring on the same invalid block number. And there was no indication mine was going to stop, I had to control c, and on restart the error went away. |
|
@Qwinn1 you are also using Paritydb? |
|
I'm afraid I don't know enough to answer that question. I haven't done anything deliberate to engage anything like that. |
|
It is Paritydb, @bkchr, that is from Subspace node and we compile it without RocksDB ever since paritytech/substrate#11537 |
|
Ah my apologies, I thought I was posting to subspace github, not polkadot |
|
Just saw it again, single occurrence: |
|
@nazar-pc I assume you don't have any concrete way for reproduction? |
|
Could you run with |
Not yet
I can, but it happens very occasionally, so no promises it'll catch anything, worth a try though. |
|
@nazar-pc do you know the block number for |
|
Okay, then I know what the problem is. By default we force canonicalize after 4096 blocks. The node is trying to import some quite block that is behind this pruning window and we end up here: polkadot-sdk/substrate/client/state-db/src/noncanonical.rs Lines 264 to 274 in ede4a36 So nothing bad is happening. However, we should improve the situation here, but I have no good idea right now. |
|
We do technically finalize blocks but the finalization point is waaay beyond 4096 blocks deep, sometimes tens of thousands of blocks deep actually. |
|
This did not reproduce with a node running for around 4 days. Do you still see this @nazar-pc on the latest release? 🙏 |
|
Not any time recently and even before that it was not deterministic |
|
Closing as too old for now |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is there an existing issue?
Experiencing problems? Have you tried our Stack Exchange first?
Description of bug
I suddenly started getting these on one of the nodes:
Interestigly, sync still works, but state database error makes me a bit concerned about what is going on here. My undestanding is that this should never really happen, hence bug report.
Steps to reproduce
Not clear yet, just happened on one of the nodes, the same machine had another node running as well and didn't observe the same issue there.
This is using ParityDb 0.4.12 if that matters.
The text was updated successfully, but these errors were encountered: