Trie refcount bug

[mikhailOK]

• Trie nodes are stored in storage with refcounts
• TrieChanges stores a delta of refcount
• StoreUpdate stores a database transaction with exact writes of refcount
• Applying TrieChanges to storage reads refcount from storage, increments it by delta and creates a write for StoreUpdate
• Writing multiple TrieChanges into StoreUpdate in a single transaction will always read refcount before the transaction, so instead of the sum of increments only one is applied.

This is most likely what causes #2500 #2544 #2556

[mikhailOK]

The plan is to fix this by separating trie nodes from different shards in storage - this will also
allow us to solve #2569.

[mikhailOK]

Turns out separating shards is not quite enough, because gc code wants to delete multiple blocks in one StoreUpdate.
The new plan is to either change gc code, or make the alternative fix (refcount-aware StoreUpdate). Discussing with @Kouprin

[bowenwang1996]

I don't understand why this cannot happen on a single shard. It seems that

Writing multiple TrieChanges into StoreUpdate in a single transaction will always read refcount before the transaction, so instead of the sum of increments only one is applied.

is not related to the number of shards.

[bowenwang1996]

On betanet I have observed a couple times that rpc nodes crashed on this line https://github.com/nearprotocol/nearcore/blob/b2d469e3da338bf933219ef351cc050075febcd0/neard/src/runtime.rs#L883 However, I have not observed crashing of validator nodes myself, even though one such incident was reported to me by one of the validators.

[mikhailOK]

@bowenwang1996 TrieUpdate is created for a chunk. Multiple chunks means either multiple shards (fixed by separating tries) or a multi-block StoreUpdate (#2613)

[bowenwang1996]

@mikhailOK is it true that #2613 is possible in a single shard?

[mikhailOK]

#2613 only leads to gc not deleting data. It's blocking because the fix to this task adds an assert that fails on #2613.