Return connection errors from the heartbeat task to the Client

[teor2345]

Motivation

We're seeing "failed servers must set their error slot" panics when Zebra is overloaded.

This panic is caused by the design of the Client connection code. Currently, the Client tasks share an ErrorSlot mutex, which must be updated before they exit.

This ticket stops using the error slot in the heartbeat task.

Designs

Instead of using the error slot, Client tasks should exit and return an error to the client instance.
The Client should exit when any task errors or stops.

We could try something like this:
(click for details)

Replace Client.error_slot with returning a PeerError:

• end the task when there is a permanent connection error, returning `Result<(), PeerError>’
• check the error via the JoinHandle in the Client
• shut down the Client when the heartbeat task exits or errors

Related Work

This is part of:

• Security: Remove the ErrorSlot from the heartbeat and connection tasks #3263

[teor2345]

Here's the code we need to return an error from:

zebra/zebra-network/src/peer/handshake.rs

Lines 1098 to 1117 in 42ef884

	/// Send periodical heartbeats to `server_tx`, and update the peer status through
	/// `heartbeat_ts_collector`.
	///
	/// # Correctness
	///
	/// To prevent hangs:
	/// - every await that depends on the network must have a timeout (or interval)
	/// - every error/shutdown must update the address book state and return
	///
	/// The address book state can be updated via `ClientRequest.tx`, or the
	/// heartbeat_ts_collector.
	///
	/// Returning from this function terminates the connection's heartbeat task.
	async fn send_periodic_heartbeats_with_shutdown_handle(
	connected_addr: ConnectedAddr,
	remote_services: PeerServices,
	shutdown_rx: oneshot::Receiver<CancelHeartbeatTask>,
	server_tx: futures::channel::mpsc::Sender<ClientRequest>,
	mut heartbeat_ts_collector: tokio::sync::mpsc::Sender<MetaAddrChange>,
	) {