Create API calls for non-blocking packet writes

[fyodor]

Npcap offers ways to read packets asynchronously, but packet writes are always blocking. We basically use the Windows version of the UNIX write() syscall. This is because we wait around to ensure the write was successful and return the appropriate error code if not. This usually doesn't take very long, but we did have one Npcap OEM redistribution license customer recently report that it could take up to 3ms and that caused problems for their real-time software.

The simplest approach would be to make all packet writes asynchronous, but that would be problematic for folks who would prefer waiting a millisecond or so to ensure the write was successful. So a better approach would probably be to offer an asynchronous option for those who want it.

We looked at how the (now obsolete) Winpcap system handled it. The system call to give the packet to the driver was synchronous, but the driver called an asynchronous NDIS function and returned immediately. It would give the illusion of being fire-and-forget. You get errors if you did something dumb with the packet structure as defined in Packet.dll API, but you don't get to find out if Windows actually sent your packet. Also the driver also did a wait for an event that the NDIS callback would set, so still it was effectively a blocking call.

Npcap has changed that: the write handler is asynchronous and indicates that to Windows by returning STATUS_PENDING. Then Windows does the work of waiting. That means the way is clear for Packet.dll to use Windows-provided synchronization routines like WriteFileEx and LPOVERLAPPED struct to do the work.

We would want to coordinate with the libpcap team--probably open an issue on their tracker to discuss how they prefer to handle asynchronous calls like this. That way we may be able to get a standardized and cross-platform solution.

[dmiller-nmap]

Some thoughts and notes from my research:

• In order to make overlapped I/O function calls on a handle, the handle has to be opened with FILE_FLAG_OVERLAPPED. This means that we need to extend/replace PacketOpenAdapter() as well as PacketSendPacket().
• Making asynchronous I/O available in a cross-platform API like libpcap is going to be tricky. Since the two examples I am familiar with (libaio and Windows) both use a struct for the async operation info, it would seem like we could make the common API just pass a void * and then define the struct it points to differently for each platform. That way Windows programmers could treat it as LPOVERLAPPED, and Linux could treat it as struct aiocb *.
• We've considered developing an alternative API (replacing Packet.dll) for a while (see Potential enhancement: alternative to Packet.dll #569), and this could be a good time to begin. On the other hand, tying this feature to that larger effort might delay the feature unnecessarily.
• A new or extended API would ideally also have sufficient extensibility to handle raw WiFi "mode," pcap_setdirection, timestamp mode and precision, and any other libpcap features that may make sense to implement in the future.

[guyharris]

and Linux could treat it as struct aiocb *

s;Linux;platforms supporting the POSIX asynchronous I/O interface;

Although that requires that the platform's mechanism for transmitting packets also support that interface. (But details such as that probably belong on the libpcap issues list.)

[fyodor]

A customer recently reported a potentially related performance issue. They are framing and sending numerous raw packets at different time intervals ranging from 20ms to 2s. They use pcap_sendpacket in one thread while capturing in an independent thread iwth pcap_loop to check whether they were delivered in the desired time. They noticed that when pcap_loop is enabled, the pcap_sendpacket call to take longer, in turn causing delayed sending of packets. They also noticed better performance on 32-bit systems than 64-bit ones.