Drain udp socket recv buffer without any significant delay

[goshawk-3]

Describe what you want implemented
As it turns out that under heavy load a kadcast peer may lose chunks (respectively messages) due to full udp_recv_buffer we should consider to drain recv socket as fast as possible.

Currently, we receive a chunk from socket, try to decode full msg (a CPU-intensive task) and then receive next chunk

  let (_, remote_address) =
                socket.recv_from(&mut bytes).await.map_err(|e| {
                    error!("Error receiving from socket {}", e);
                    e
                })?;

            match Message::unmarshal_binary(&mut &bytes[..]) {
                Ok(deser) => {
                    trace!("> Received {:?}", deser);
                    let to_process = decoder.decode(deser);

Instead we could try to process/decode the received chunk asynchronously.

Pseudo idea:

loop {
 # recv_from a socket
 socket.recv_from(&mut chunk).await.map_err(|e| {
                    error!("Error receiving from socket {}", e);
                    e
                })?;

 # run fast sanity check, drop and report if invalid

 # delegate it to another task
 channel.send(chunk)
 
}

Describe "Why" this is needed
This should mitigate the pressure on OS-level udp_recv_buffer and give us more control over chunk buffering. That's said, we'll be able to trace/report when a chunk is lost, resize buffer/chan size dynamically if needed.

Describe alternatives you've considered
In addition to this idea, a kadcast peer will be able to set udp_recv_buffer at socket level (SO_RCVBUF). However this might be limited due to OS-level net.core.rmem_max config.

Additional context
Currently, a system-test with following params CLUSTER_SIZE=10 MSG_SIZE=1000000 WAIT=5s go test -count=10 -tags=testbed -run TestCluster on Linux will pass only if net.core.rmem_default is set to a large number.