Finalized Deposit Trie Is Cached Incorrectly

[nisdas]

🐞 Bug Report

Description

Currently the proposer falls back to rebuilding the whole finalized deposit trie if we do have an invalid root for it. This was introduced in #8869 . The fallback was a workaround to the possibility that our cached trie had some inconsistent insertions leading to an invalid root. It was decided that falling back is better than potentially producing a block with an invalid deposit root and any resulting proofs from it. However as this comment #8869 (comment) alludes to, if the deposit trie takes too long to rebuild this can lead to block proposals being missed completely. The possibility that a block proposal can be missed from hitting this edge case is a good enough reason to ensure that the root cause for the bug is found and fixed. This issue is to track the core bug and the eventual resolution for it.

Has this worked before in a previous version?

No

🔬 Minimal Reproduction

This is hard to reproduce and has only come up in rare occasions.

🔥 Error

Cached deposit trie is invalid, rebuilding it now

What version of Prysm are you running? (Which release)

v2.0.5 and before