Introduce an in-order processing mode for WriteRequests

[jonathan-dilorenzo]

I'm hoping to introduce different processing modes for write requests. Right now, the only processing mode is Arbitrary Order (or something like that), which denotes that the server can process the updates of a write request in any order it wants.

We're hoping to introduce an In order processing mode, denoting that the server must process the request from start-to-finish. The goal, for us, is that this change (along with a new atomicity mode that I'm hoping to introduce, which is basically, STOP_ON_ERROR) will let us send a bunch of updates where we care about the ordering in a single batch, which is a more significant gain for us than letting the switch choose its processing order. Of course, this also means that the performance of a request is likely dependent on ordering in a way that it didn't use to be, potentially putting a new onus on the controller to understand what leads to high vs low performance.

I'm planning to make a PR for this to make it more concrete, but am currently having some GitHub access issues that I'm working on resolving.

[smolkaj]

Any updates on this issue? Should we discuss this at the WG meeting tomorrow?

[jonathan-dilorenzo]

We can, but I'm prioritizing PR #472 before next meeting, so I'm not convinced there will be much to say. I believe the last update here was for me to decide which one I liked among a number of suggested designs and write a PR for that so we can discuss more concretely. The most promising designs were:

1. Introduce a 'barrier' update such that any updates before the barrier must be attempted prior to any updates after the barrier. This barrier would likely have an enum or boolean determining whether writing should progress past the barrier or stop on failure. There is some concern whether this would work well in our use-case (which is FIFO).
2. Add a processing mode and an additional atomicity mode as per comment#1.

Another option is to encode the barrier with another level of indirection, though this would be slightly trickier to make backwards compatible. And we've had discussions about how there are already too many levels of indirection for use cases that write large amounts of updates.

[smolkaj]

@jonathan-dilorenzo, would we want to push on this before 1.4.0? If so we should add the label.

[jonathan-dilorenzo]

I think let's not make that assumption at least. If 1.4.0 ends up getting delayed sufficiently, then I may be able to, but I don't really have the cycles in the next few months at least.

[chrispsommers]

@jonathan-dilorenzo bump - Sept 13 cutoff for 1.4.0

[jonathan-dilorenzo]

Thanks for the ping Chris! Let's plan not to get this in for 1.4.0. I think it needs a bit more thought.