WARN [...] GC Error: block on canonical chain shouldn't have refcount 0

[tifrel]

Describe the bug
Garbage collection warnings when running a mainnet validator node.

2024-08-12T20:48:11.274876Z  WARN garbage collection: Error in gc: GC Error: block on canonical chain shouldn't have refcount 0

To Reproduce
Run a 2.0.0 validator on mainnet, freshly set up from data backup, tracking shard 0 with mem tries loaded into memory. Machine specs just at the minimum (8C/16T, 80 GB RAM, 800 GiB/s SSD)

Expected behavior
A clear and concise description of what you expected to happen.

Version (please complete the following information):

• nearcore tag 2.0.0
• rust version 1.78.0
• not using docker
• mainnet

[telezhnaya]

From the author:

Created the issue, and it seems to be correlated with machine specs. I set up a new node with ~150% minimum specs (15C/30T, 120 GB RAM, RAID 0 SSD Array). No warnings/errors so far, will monitor and update.

My suggestion: let's upgrade the minimum requirements in our documentation and in the release notes

[telezhnaya]

The requirements have been updated

[wackazong]

@telezhnaya I have a top 100 validator. I am currently trying to get a new backup node started. I followed the requirements for the new backup node scenario, downloaded a new snapshot, and I get the same behaviour. My machine has 15 Cores, 32GB RAM and four parallel fast SSDs in RAID0.

What can I do to help analyse this issue?

[evgenykuzyakov]

@tifrel Do you know which snapshot you've used?