Order of retransmitted messages

[marygilliland]

I am working with the example programs normDataSend and normDataRecv distributed with the norm library.

When I use NormSetTxLoss() to force dropped messages for testing purposes, I see the receivers send NACKs and later receive the dropped messages. However the dropped messages are received out of order. For instance, if the expected order is A, B, C, D, but the message B is dropped, then I see the receiver get A, C, B, D or A, C, D, B.

Is there some setting or technique to keep the messages in the original order? I thought I saw something indicating that the original order can be maintained, but I haven’t been able to find it again.

Thanks.

[bebopagogo]

The NORM_OBJECT_DATA form of transport treats each content "object" as an atomic unit and the NORM API code delivers those received objects to the application as soon as they are received without regards to order of delivery. The idea here is some applications may be able to make use of the NORM_OBJECT_DATA content right away (i.e., with lower latency) even if it is out-of-order with respect to how the sender enqueued the data. Note that NORM_OBJECT_STREAM, however, does enforce in-order delivery of data (just like TCP does) and you can use the NormSetEom() marking to designate "message" boundaries within the stream if your application is sending "messages" as opposed to a byte-stream. NORM_OBJECT_STREAM can have some efficiencies over NORM_OBJECT_DATA for applications with "small" messages since the NORM FEC erasure code blocks can span multiple "messages" where that erasure coding is limited to the context of a single NORM_OBJECT_DATA "message" when that transport mode is used.

BTW, I recommend looking at the norm/examples/normMsgr.cpp (there are also normMsgr.py and NormMsgr.java versions of this example) as a more complete NORM_OBJECT_DATA example as it illustrates a number of addtional concepts. Additionally the norm/examples/normStreamer.cpp is a fairly functionally equivalent for normMsgr but uses the NORM_OBJECT_STREAM transport type.

If you really want to use NORM_OBJECT_DATA, then your application will need to enforce ordering itself. I could add an API call that would let you access the NORM "transport object id" for the received NORM_OBJECT_DATA items and you could use that as a helper to enforce your ordering. However, those "transport object ids" do eventually wrap for a given NormSession or can be reset if a sender exits and rejoins a multicast group, so you would have to take care in doing that. If you are not concerned about absolute overhead efficiency, using your own message id would be more straightforward.

Hopefully this is helpful to you.

[marygilliland]

Thank you for your reply.

I did initially investigate the NORM_OBJECT_STREAM form of transport, but when I found that it was "bundling" application messages, I changed to trying the OBJECT_DATA transport. I thought that the bundling behavior would affect timing of application messages and that was not desirable. I do see what you are saying about the efficiency aspects related to FEC behavior. But is there something in the API that would allow turning off that bundling behavior, or kind of "overcoming" it, maybe something about flushing, or the buffer size that the application sets?

And thank you for the offer of an added API call. I definitely need to do more testing before requesting something like that. I have looked at the other examples you mentioned and have been starting to try some of those additional behaviors after learning from the more simplified examples.

[bebopagogo]

Yes, you can override the stream "bundling" by using the NormStreamFlush(streamHandle, true) call. This causes NORM to immediately send out a packet for the stream even if it is a small packet with less payload size than the configured segment size (The "true" argument also marks the end of message boundary). This should be called after all the bytes for the message you are trying to send have been successfully written to the stream via NormStreamWrite(). There is also an option to enable "auto flush" for a stream that automates this action whenever a NormStreamWrite() call accepts all of the bytes you provide to it (I'm not sure I recommend using that option unless it provides a notable performance advantage (due to fewer NORM API calls) for a high speed use case.

So if you use NormStreamFlush() and your messages are smaller than the set NORM segment size, there will be a one-to-one correspondence of NORM_DATA packets and your messages. Note that when you use NORM message boundary marking via NormMarkEom() (or the above "true" argument of NormStreamFlush()), a late-joining receiver can "find" message boundaries mid-stream by using the NormSeekMsgStart() call.

The "normStreamer.cpp" example code illustrates all of this stuff. I have also used it for text chat messaging, audio and video streaming applications where it's acceptable for a receiver to join a multicast (or unicast for that matter) session already in progress.

[marygilliland]

Thanks for this info! I have my application calling NormStreamFlush(streamHandle, true) after writing all bytes of the application message via NormStreamWrite(), and it's working just as I need it to.

[marygilliland]

I'm doing further testing with this application, attempting to run with the receiver in silent mode and the sender sending auto parity packets. The sender has block size set to 64 and parity set to 16, and the auto parity also set to 16. I'm using the flush command so that each application message is treated as a block. The application is set to send 100 application messages and then do nothing more except continue to watch for NORM events. Via the trace messages, I see the first 64 application messages (blocks) get sent and then the 16 auto parity messages. Then the remaining 36 application messages get sent, but no auto parity messages are sent. If the receiver experiences a loss among those first 64 application messages, it recovers the loss when the auto parity packets are sent. But if it experiences a loss in the second group of application messages, it doesn't receive any more messages after that lost one - because it's waiting for the repair of that lost one so that all messages can remain in the order sent. I'm calling NormStreamClose() with the graceful flag set after sending the 100th application message, which I expected to inform Norm that the application doesn't intend to send any more messages. Following the stream close, the application gets a TX_OBJECT_PURGED event, and I’m calling NormStopSender() at that point. What am I missing?

Thanks for your help in understanding the use of this API.