network/libp2p: worker CPU usage increases from v1.13.0 to v1.15.0

[alexggh]

Our kusama validator have been upgraded from v1.13.0 to v1.15.0 and since then we see a dramatic increase in CPU usage:

Dashboard: https://grafana.teleport.parity.io/goto/OgvYN59IR?orgId=1
The network seems to be fine, but we should aim to decrease the cpu usage.

The number of messages seems to have stayed the same and the entire networking increase seems to come from libp2p-node task.

Note! there seems to be a small increase 5% in the cpu usage for litep2p, but not as dramatic as we see for libp2p-node, so my bet is that this might come from the libp2p upgrade we performed in v1.14.0: #1631.

cc @paritytech/networking
FYI @sandreim, @AndreiEres

[dmitry-markin]

I can confirm that networking CPU usage increased 2x after libp2p upgrade. Here are the results for versi network running 100 validators with libp2p-0.51.4 and 5 validators with libp2p-0.52.4:

The node from commit 3c21372 (the latest commit with libp2p-0.51.4)

The node from commit 414a8fc (the next commit from master, libp2p-0.52.4 upgrade)

The results for other nodes on the network from the same commits are consistent with average load within +/- 1%.

Even though the plots are quite noisy, the average networking CPU load definitely went 2x from 6% to 11%.

[alexggh]

Any idea if the CPU increase leads to gains in other dimensions, like throughput/latency ?

[lexnv]

After looking at litep2p vs libp2p nodes side by side, it seems that libp2p is establishing more connections than litep2p with connections limits.

Last time I checked, inbound connections are rejected only after establishing (ie negotiation).

I would add a limit in libp2p of 1k or lower to pending connections as well. We currently only limit establised (negotiated).

[dmitry-markin]

Any idea if the CPU increase leads to gains in other dimensions, like throughput/latency ?

Nope. At least nothing points at it in the libp2p changelog.

[dmitry-markin]

The proper way forward with libp2p would be IMO upgrading it to the latest version, and if we still have the issues with CPU load - profiling and fixing libp2p. Unfortunately, even the upgrade of libp2p is already quite time consuming.

We can still introduce the limit @lexnv proposing, as this could mitigate the issue in some setups.