Consensus based sequence

[SampoSyrjanen]

Hello, I'm trying to implement a sequence (append-only) where each client would get the same sequence and events in the same order. I believe this should be possible in fluid because of the sequencing of operations.

I see that there is an implementation of a consensus based queue. However, I don't understand how to use it. It seems you can add and remove (acquire) items from it, but is it possible to get the contents? Does the "add" events come in the same order for every client? I'm also experimenting with the task manager to exclusively add items to another DDS to get the desired behavior. But this feels a little roundabout way of implementing this.

Any comments and help is appreciated. Thanks.

[alexvy86]

where each client would get the same sequence and events in the same order.

This is one of the tricky bits. Clients that issue ops normally apply them locally right away, which strictly speaking means they see them "out of sequence" compared to when every other client receives the same op after the server sequences it. When the sequenced op comes back to the client that issued it, I think it usually ignores it if it was sequenced/acknowledged by the server because it was already applied locally and the merge logic of DDSes should ensure that the end result will be the same as the one other clients see even though they potentially applied this particular op in a different order. If that scenario is not an issue, then I think default Fluid behavior should guarantee that ops are applied by all clients in the same order (not necessarily the same time though) and thus events also fire in the same order.

Otherwise, it might be an option to create a DDS that does not apply ops locally right away, but waits until they come back sequenced from the server. Then every client would see them in the exact same sequence. I'm not the expert on the area though, so not entirely sure if this is possible.

[tyler-cai-microsoft]

Otherwise, it might be an option to create a DDS that does not apply ops locally right away, but waits until they come back sequenced from the server. Then every client would see them in the exact same sequence. I'm not the expert on the area though, so not entirely sure if this is possible.

To add on to what @alexvy86 said, it is possible, but it depends on your requirements.

There are drawbacks with going with a remote only solution where the changes only appear once they're sequenced. Some include:

• Special handling for offline, disconnects, and reconnects. This of course isn't a problem if you only expect this DDS to work when you're online.
• Changes appear slower as they need to round trip through the service
• There are more considerations as well, including fluidhandles (which means you will need to worry about GC)

[tylerbutler]

ordered-collection and register-collection seem relevant.

[pk-pranshu]

@SampoSyrjanen - Can you please share the details on the scenario that you are targeting? We would like to see if there is a better path forward for your team here.

[SampoSyrjanen]

Thank you for the replies. I made a small test project to illustrate how sequence event order can be the same for all clients if the task-manager DDS is used to coordinate inserts.

https://github.com/SampoSyrjanen/fluid-exclusive-inserts

The sequence values can be just plain JSON values. The implementation is expected to be slower than default fluid behavior.

The ordered-collection seems very relevant indeed, but I cannot find any examples for that. How to subscribe for "add" events and how to read the contents of the queue?

[SeanSnyders]

@pk-pranshu, any input on the information @SampoSyrjanen provided?