[MAINT] Revise the flow window, bufferleft and application congestion response rules

[ethouris]

The value of ACKD_BUFFERLEFT reported via ACK message saved into m_iFlowWindow field is used to determine if the sender is free to go with sending a "unique" packet (retransmitted packets are being sent anyway). There are few problems:

1. The ACKD_BUFFERLEFT is updated with the value that designates the "unacknowledged" range in the receiver buffer, not "free" range. The unacknowledged range is the range of packets that are ready to be retrieved by the application just after the ACK has signed them off. If then there is some further range of packets that start from at least one loss, it is included in this number, while it plays no role in the free space for newly sent unique packets.
2. The "flow window" term is not exactly defined. We have the following qualification of packets:
   • Flight: packets that have been sent, but not yet arrived. That term is also floating due to the fact that you can't take a measurement snapshot on two machines at the same time and interpret together on one of them. Therefore it can be at best defined as the distance in sequence numbers between the packet that has been reported as received exactly at the moment when forming the ACK message; that could be equal to the acknowledgement sequence if there are no lost packets.
   • Flow: packets that have been sent, but not yet acknowledged. It may enclose packets that did arrive at the receiver (so they are not "in flight") but may follow lost packets, which might be defined with a different term - let's say "recovery". This way then Flow = Flight + Recovery.

It should be then decided exactly what should be measured and for what the measured value should be used.

For example: if we want to decide whether the unique packet can be sent, that is, whether the receiver has some space to store it and will not have to discard it, then the sender requires the information from the receiver regarding the free space in the buffer following the last received packet because this exactly defines the free space for the unique packets (the space between ACK and the last received is reserved only for retransmission). Note that the Recovery Window, which influences the value of the Flow Window, may have random values, potentially from 0 up to the size of the receiver buffer, hence including this value in the condition deciding about whether to send a unique packet is misleading. For example, if we have 8192 total size of the buffer, the number of sent packets were up to position 8180 in the receiver buffer, and there would be 10 packets in flight, but the packet at position 3000 has been lost, then a packet aiming at position 8191 will be met with the reported flow window of 5192 packets, while in fact the free space left for sending packets is 1, which is already less than the flight window and should cause sending of the unique packets to be stopped. At least with the reported value of 12 and appropriately treated measurement of the flight window (which also isn't being done - and it could be, for example, by providing the sequence number of the last received packet in the ACK data) it should find this value already as a stop condition.

Note that the flight window term used in TCP defines this way all unacknowledged packets, but TCP also disregards the recovery window completely. Packets that came after a loss are just a noise for TCP, and therefore there's no difference between "retransmission" and "unique" packets. Double ACK or exceeding the flight window makes TCP simply go back to the ACK position and start over, and at this moment the flight window gets reset to 0, growing with every packets sent (again) and shrinking with ACKs.

In SRT (UDT) it has never been a case because a received packet is a received packet, whether in order or not, and retransmitted are only lost packets. Hence retransmitting a packet is something else than sending a unique packet, and for a retransmitted packet there will always be a space to accept it (as long as its sequence isn't in the past, that is, it was already discarded).

There's also another issue that is there since UDT times and with received 0 in the ACKD_BUFFERLEFT field it fixes it to 2, with an explanation in the comment that it would otherwise cause a deadlock. This is likely some old bug or misconception in the UDT that has never been fixed:

        data[ACKD_BUFFERLEFT] = (int) getAvailRcvBufferSizeNoLock();
        // a minimum flow window of 2 is used, even if buffer is full, to break potential deadlock
        if (data[ACKD_BUFFERLEFT] < 2)
            data[ACKD_BUFFERLEFT] = 2;

This makes that sending is continued even if it is useless. Of course, this still may be less than the flight span and still prevent unique packets from being sent, but such a fixing seems like a problem in the design. What is expected in a situation when the receiver buffer is full (the application doesn't purge it on time), is that the sender should stop and wait for the update from the receiver. But in order this to work, it must be possible to send ACK when the time comes even if ACK would come with exactly the same sequence number - but it might still come with a different value in ACKD_BUFFERLEFT, and unblock the sending.

To allow this, there must be also somehow ensured that the buffer is also big enough to cover the flight window, or maybe even the whole flow window, and the measured flight window occupies only a fragment of the buffer space. Some fallbacks might be thought of in case when this isn't satisfied, with appropriate criteria and limits (for example, if a flight window has been measured as taking more than 50% of the buffer space, sending must be slowed down, even if the link capacity is way higher). The flight window should be measured in order to have idea, how many less packet space is in the receiver buffer towards the reported value at the moment of ACK reception, and the decision of stalling and resuming the transmission should be based on that value, possibly also configured with the fixed flight window size and safety margin for the measured flight window size. As well as both flight window and recovery window should be measured separately, while the recovery window should be the value used to shape the sending period. Might be also that the information about how many packets are lost in the recovery window (as a bitmap of first 32 packets following the ACK) could be useful, although the loss information should be still in the loss reports.

[maxsharabayko]

Related issue #700.