Gossip proofs should be ingested into blockstore's, and used to notify ReplayStage of duplicates

[carllin]

Problem

Gossip proofs are propagated but not currently:

1. Ingested into blockstore's duplicate column family (store_duplicate_if_not_existing() only runs on received shreds from WindowService)
2. Not used to notify ReplayStage of duplicates

This is an issue if some nodes see duplicates and others don't.

Proposed Solution

[t-nelson]

what's the size and rate of the proofs?

cc/ @behzadnouri

[wen-coding]

Oops, closed by accident, still need to add protection.

[behzadnouri]

what's the size and rate of the proofs?

The size is equal to 2 shreds, but because there are some metadata it spills over 3 packets.
The rate they are pushed to gossip can be monitored by these metrics:
https://github.com/solana-labs/solana/blob/b36791956/gossip/src/cluster_info_metrics.rs#L632-L633

[behzadnouri]

@wen-coding This is not being done yet

and used to notify ReplayStage of duplicates

[wen-coding]

The plan ahead: Ashwin already has PR #32963 to notify state machine of duplicate proofs, but this can't be done until we fix the switching proof. So roll that out after the switching proof is fixed. @AshwinSekar Does that look right?