statement-distribution high cpu usage

[sandreim]

This looks like a bug, the nodes aren't even in active set and still we see this usage pattern:

https://grafana.teleport.parity.io/goto/UfZem2JIR?orgId=1

[alexggh]

Interestingly enough after nodes have been restarted for upgrading to 1.9.0, statement-distribution looks unused.

This feels like a small leak, unfortunately we don't have metrics going back, I'll keep on eye on the metric over-time to confirm it.

[alexggh]

This seems to have happened again on one of our kusama validators after a few weeks

I don't think it is a leak, looking at this dashboards it seems like we are doing a lot of polling somewhere, notice how the average poll time actually goes down and the number of them increases.

I suspect something is off with this next_retry and wakesup very often

polkadot-sdk/polkadot/node/network/statement-distribution/src/lib.rs

Line 130 in 76719da

let retry_request = v2::next_retry(request_manager).fuse();

,
although I don't have the data to back it up nor did I found any logic error that might get us into this state.

[alexggh]

And the fix: https://github.com/paritytech/polkadot-sdk/pull/4192/files

[sandreim]

Fixed by #4192