Use MPMC bounded queue for group freelist

[eugeneia]

This adresses

• Group freelist contention #1468

by employing a smarter data structure for the group freelist (MPMC bounded queue).

It also moves the rebalancing house keeping into packet.free, and keeps limits on the upper bound of work performed in any rebalance/reclaim step.

In the density plots below we have master in red and this branch in blue (green was an alternative wip branch).

This certainly improves latency and thus performance of interlinks, however this does not make interlinks scale without restriction. The benchmarking I did seems to show that sharing memory between cores still turns into a bottleneck, and depending on your CPU architecture you are going to run into that sooner or later. I did compare results between Intel and EPYC machines and they are quite different. But for here and now I’m going to focus only on EPYC as an example.

In the plot above we compare latencies by number of receivers for a single transmitter, and we see a significant blowup of latencies after 1 transmitter + >5 receivers. Now why is that?

If we look at the topology of our CPU as reported by AMDuprof we can get a hint:

CPU Topology:
Socket, CCD, Core(s)
0,0, 0 1 2 3 4 5
0,1, 6 7 8 9 10 11
0,2, 12 13 14 15 16 17
0,3, 18 19 20 21 22 23

Each CCD spans six CPU cores. So while our workload fits a single CCD we get OK perf (~60Mpps) but as soon as we add a receiver running on a distinct CCD latency and perf tanks (~10Mpps).

The above diagram of the CPUs architecture/topology gives some hints. So each CCD houses two CCXs, and a cores in a CCX share a L3 cache. I am assuming the CCXs in a CCD also have faster interconnects to each other than to a CCX in a remote CCD?

Anyways if I look at some PMU counters using AMDuprof we can maybe see why a workload distributed across CCDs fares worse (take this with some salt, this is me reading the tea leaves):

• workloads that fit a single CCD can fetch data from shared L3 or other L2 caches in the same CCD or CCX? DCFillsFromL3orDiffL2 is higher
• they incur less L2DtlbMisses, and have to perform less DCFillsFromLocalMemory
• hence they can retire more instructions per cycle