Make sure nodes don't hammer each other even when reserved

[tomaka]

Right now, when node A has node B as reserved, node A will try really hard to open a substream to node B.
Even if node B refuses the substream, node A will almost immediately try again. This results in a spam of substream open attempts, which unnecessarily uses bandwidth and CPU.

If we have a lot of reserved nodes, for example because of the collation and validation substreams in Polkadot, and most of them refuse the substream, this spam can be considerable. They're not supposed to refuse the substream in normal situations, but with our recent revert to 0.8.30, they do in practice.

This PR adds a 10 seconds "ban" during which we don't try to re-open a substream after it has failed to open, even if the node is reserved.

[kpp]

Is there some kind of dynamic backoff with randomness? If all nodes got disconnected, we will spam with requests exactly in 10 seconds.

[tomaka]

Indeed, I changed it to a random 5 to 10 seconds interval. We actually use randomness everywhere else already.

[andresilva]

bot merge

[ghost]

Trying merge.

[burdges]

Why a fixed or random back off instead of a progressively larger one?

[rphmeier]

I foresee this potentially causing some issues with collator networking. If a validator initially refuses a connection from a collator due to a race in when they see the next relay chain block, then the collator won't attempt to connect for 10 seconds. That may lead to unnecessary gaps in the parachain.

[kpp]

@burdges, @rphmeier, as I see both or your suggestions can be fixed with an exponential backoff starting from, e.g. a rand 1-2s, insreasing by a multiplier of 2^N?

[tomaka]

If a validator initially refuses a connection from a collator due to a race in when they see the next relay chain block

Validators never refuse substream from collators unless they're full.
They could accept the substream and then immediately close it, but that's a different code path.

[rphmeier]

Ok, that is true. The way that @Lldenaurois is designing the collator protocol is to maintain a peer-set which is larger than the actual reservoir of peers we accept. As we add PreVF handshake logic we'd need to maintain that guarantee, so we don't reject connections outright.