Avoid throwing exceptions in mini protocols

[edsko]

From @coot :

The network-mux library, for some time, is capable of returning values from each mini-protocol. We would like you guys to move from throwing exceptions (which was necessary in the previous version of mux) to gracefully finish a protocol and return. AFAIK that's only important for the chain-sync client when it decides that the remote chain is too old, in which case now it should not throw, but gracefully return (i.e. reach the termination state of the protocol). A clean exit allows us to restart mini-protocols on the same bearer (what the peer-to-peer governor will do when it decides to promote the peer back as a hot one).

I think you don't need to worry about synchronisation between all the protocols, e.g. if chain-sync exists, the governor should ask all the other protocols to terminate (I need to review if it actually does it though), and it's probably the best place to handle that.

This should work with the current mux compatibility layer that we are using. The small change that we'd need to do is to handle the return value and run ErrorPolicies on it to actually suspend the peer. This means this could go to master. Alternatively, it could be just merged with coot/connection-manager, and go to master with all the other changes.

When throwing is ok? When a protocol is violated. One thing to remember is when one throws from the context of a protocol it will always bring the bearer down, and since we will run it in bidirectional mode it will shutdown both responder and initiator whether the exception was thrown from initiator or responder side.

[mrBliss]

@coot Let me know by when you need this.

[coot]

Could you prioritise it? I am about to start to integrate connection-manager & p2p-governor with consensus (in a branch). These changes will need to be merged to master together. It will likely take me some time to test it, before merging it to master.

[coot]

@mrBliss the coot/connection-manager branch is integrated with ouroboros-consensus. Please open a PR against it.

[mrBliss]

Can be closed as #2525 was merged.