Too many TCP connections open when authority-discovery is enabled

[tomaka]

While we are not "leaking" TCP connections anymore, the number of connections doesn't go below 200/300 when the authority discovery is enabled.

This should be figured out.

[tomaka]

Here are some details.

The following metrics have been pulled from nodes running on a recent commit of the Polkadot master branch.

Every ten minutes, the number of libp2p tasks (in orange below) currently running on an average node increases to around 3000:
(note that the graph uses a logarithmic scale)

Unsurprisingly, the CPU consumption of the network worker task (in red) peaks every 10 minutes as well:

And the number of times calling NetworkWorker::poll takes more than one second also peaks every 10 minutes (network worker in blue):

While I don't have a formal proof that these come from the authority discovery, the only thing that "ticks" every ten minutes, as far as I know, is the authority discovery.

[arkpar]

@mxinden could you take a look?

[tomaka]

Additionally, here is the number of active TCP connections on an average node:

In red and green, the established connections. In bright red, the number of pending attempts. In yellow, the number of connections that have recently been closed.

If you except the 150 slots used for gossiping (I think our nodes are configured with 150 slots, not totally sure of that number), all the other connections are used for Kademlia purposes.

I opened this issue because I believe that this number is a bit high, but maybe this is a normal number by design, or by a bug somewhere that leaves connections open.

Tackling this issue consists in figuring out whether this is normal, and if not, decrease the number.

[tomaka]

Might be fixed by libp2p/rust-libp2p#1698

[tomaka]

I believe this is now fixed. See the description of paritytech/polkadot#1807 (comment) for details.