core: validate chain before writing block to avoid race condition

[manav2401]

Description

This PR is to avoid a race condition explained below.

Any incoming chain is checked against the local whitelisted milestone/checkpoint entries during the initial validation. We don't perform any extra checks once the blocks are executed locally. Hence, there is a possibility of receiving a milestone until the time we received the chain to import till the time we execute and write the blocks to db.

E.g. if the last milestone received was 100. We imported a chain from 101 to 120 which is an honest fork. A milestone is being generated from 101 to 116 (can differ also) but it hasn't yet reached bor. For bor, the last milestone is still 100 but 116 will be correct and whitelisted soon (from heimdall). Meanwhile, we receive a side chain (which seems to be incorrect) from 101 to 120. Because bor still doesn't know if milestone with 116 end block is finalised so it can't comment on the validity of that chain. It will try to import it normally and will pass the initial validation. Now, while the blocks of this side chain are executing, bor imports the 101-116 milestone (this is possible because canonical is still the correct chain we received initially). Now, the latest milestone is 116 (corresponding to correct fork) but our local head has the incorrect side chain and we can't know it until we receive the next milestone (even that's tricky because that will only cause a rewind till 116). Hence, the finality API will fallback to using checkpoint for returning the latest finalised block (because the block hash of whitelisted milestone and the local chain doesn't match). For an end user, this may feel like a finality violation but it's just the nature of occurrence of events which is causing it to use a fallback value (i.e. checkpoint).

Changes

• Bugfix (non-breaking change that solves an issue)
• Hotfix (change that solves an urgent issue, and requires immediate attention)
• New feature (non-breaking change that adds functionality)
• Breaking change (change that is not backwards-compatible and/or changes current functionality)
• Changes only for a subset of nodes

Breaking changes

Please complete this section if any breaking changes have been made, otherwise delete it

Nodes audience

In case this PR includes changes that must be applied only to a subset of nodes, please specify how you handled it (e.g. by adding a flag with a default value...)

Checklist

• I have added at least 2 reviewer or the whole pos-v1 team
• I have added sufficient documentation in code
• I will be resolving comments - if any - by pushing each fix in a separate commit and linking the commit hash in the comment reply
• Created a task in Jira and informed the team for implementation in Erigon client (if applicable)
• Includes RPC methods changes, and the Notion documentation has been updated

Cross repository changes

• This PR requires changes to heimdall
  • In case link the PR here:
• This PR requires changes to matic-cli
  • In case link the PR here:

Testing

• I have added unit tests
• I have added tests to CI
• I have tested this code manually on local environment
• I have tested this code manually on remote devnet using express-cli
• I have tested this code manually on mumbai/amoy
• I have created new e2e tests into express-cli

Manual tests

Please complete this section with the steps you performed if you ran manual tests for this functionality, otherwise delete it

Additional comments

Please post additional comments in this section if you have them, otherwise delete it