(RFC) Improve network bandwidth by using shared memory between tender and bindings for hvt

[niks3089]

Summarising the changes done:

Use shared memory and ring buffers provided by Muen's shmstream between hvt tender and hvt binding to speed up network bandwidth and reduce VM exits. Changes are implemented and tested only on Linux. The network performance is now almost equal to that of Qemu's.

Tender:

1. --shm argument is added for the net module to set up and use shmstream
2. IO thread is setup which is responsible to read and write packets between shmstream and tap device.
3. IO thread is completely event driven and the only sane choice was to use epoll.
4. Event fds are used to notify the following events:

• Notifying tender to read the packets from shm
• Notifying binding to read the packets from shm
• Notifying tender when the shm is XON'ed
• Notifying binding when the shm is XON'ed

5. Packet rate is flow controlled. I,E in case of read path, the packets are not read from the tap device if shmstream is XOFF'ed.

Bindings:

1. Modify the net to setup the shared memory when the option is set
   2 new APIs to do a bulk write to reduce the VM exits:
2. solo5_net_queue(): Queues the packet in shared memory
3. solo5_net_flush(): Notifies tender to read the packet from shared memory and write to the tap

Shmstream changes:

1. Provide a way for the writer to know if the shm region is full to avoid overruns.
2. Provide the ability to do flow control in shmstream using XON/XOFF.
3. When the writer detects that there is no more space available, the region is XOFF'ed. I.E, no more writes can happen.
4. The region is XON'ed again by the reader when the available space is x% of the total space allocated. Currently x is hardcoded to 20.

[mato]

Sorry for the very belated reply. I had no idea you were still working on this!

After discussing with @djwillia and @ricarkol, I'm retitling this as an "RFC" for now. I definitely want something like this for the "hvt" tender, but have no time to look at it in detail right now, sorry.

Having said that, the main points that stand out as in need of addressing are:

1. The Solo5 network API (as defined in solo5.h) should not need to change, or, if that is not possible then we need to work out some kind of compromise where the API changes required are common to all bindings/tenders. In other words, the application should not need to know or care what the underlying implementation is doing (shmstream-based or not). If that means we need to discuss a review of what a "better" network API should look like, then we should have that discussion first.
2. If possible, I'd start with a minimalist version that uses shmstream but does not necessarily introduce I/O threads or other complexity in the tender even if that comes with the cost of (some) performance loss. Once we have that working, only then decide what extra complexity is worth some extra performance, since that complexity (eventfd, threading, etc.) is not "free" from a simplicity and security point of view.

I realise that you are now probably busy with other things. So, is it OK with you to use your changes as "inspiration" for a future Solo5 release / re-work them into something upstreamable?